Horton
Menu

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Explain about Column List feature.

  2. Doc: fix column list vs. replica identity rules.

  3. Prohibit combining publications with different column lists.

  4. Fix the check to limit sync workers.

  5. Wait for subscription to sync in t/031_column_list.sql

  6. Move prattrs to the pg_publication_rel section in docs

  7. Allow specifying column lists for logical replication

  8. Fix row filters with multiple publications

  9. Fix publish_as_relid with multiple publications

  10. Add some additional tests for row filters in logical replication.

  11. Add index on pg_publication_rel.prpubid

  12. Avoid using DefElemAction in AlterPublicationStmt

  13. Small cleanups related to PUBLICATION framework code

  14. Add PublicationTable and PublicationRelInfo structs

  15. Fix various concurrency issues in logical replication worker launching

  1. Column Filtering in Logical Replication

    Rahila Syed <rahilasyed90@gmail.com> — 2021-06-30T19:36:11Z

    Hi,

Filtering of columns at the publisher node will allow for selective
replication of data between publisher and subscriber.  In case the updates
on the publisher are targeted only towards specific columns, the user will
have an option to reduce network consumption by not sending the data
corresponding to new columns that do not change. Note that replica
identity values will always be sent irrespective of column filtering settings.
The column values that are not sent by the publisher will be populated
using local values on the subscriber. For insert command, non-replicated
column values will be NULL or the default.
If column names are not specified while creating or altering a publication,
all the columns are replicated as per current behaviour.

The proposal for syntax to add table with column names to publication is as
follows:
Create publication:

CREATE PUBLICATION <pub_name> [ FOR TABLE [ONLY] table_name [(colname
[,…])] | FOR ALL TABLES]


Alter publication:

ALTER PUBLICATION <pub_name> ADD TABLE [ONLY] table_name [(colname [, ..])]


Please find attached a patch that implements the above proposal.
While the patch contains basic implementation and tests, several
improvements
and sanity checks are underway. I will post an updated patch with those
changes soon.

Kindly let me know your opinion.


Thank you,

Rahila Syed

  2. Re: Column Filtering in Logical Replication

    Dilip Kumar <dilipbalaut@gmail.com> — 2021-07-01T06:20:36Z

    On Thu, Jul 1, 2021 at 1:06 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
>
> Hi,
>
> Filtering of columns at the publisher node will allow for selective replication of data between publisher and subscriber.  In case the updates on the publisher are targeted only towards specific columns, the user will have an option to reduce network consumption by not sending the data corresponding to new columns that do not change. Note that replica identity values will always be sent irrespective of column filtering settings. The column values that are not sent by the publisher will be populated using local values on the subscriber. For insert command, non-replicated column values will be NULL or the default.
> If column names are not specified while creating or altering a publication,
> all the columns are replicated as per current behaviour.
>
> The proposal for syntax to add table with column names to publication is as follows:
> Create publication:
>
> CREATE PUBLICATION <pub_name> [ FOR TABLE [ONLY] table_name [(colname [,…])] | FOR ALL TABLES]
>
>
> Alter publication:
>
> ALTER PUBLICATION <pub_name> ADD TABLE [ONLY] table_name [(colname [, ..])]
>
>
> Please find attached a patch that implements the above proposal.
> While the patch contains basic implementation and tests, several improvements
> and sanity checks are underway. I will post an updated patch with those changes soon.
>
> Kindly let me know your opinion.
>

I haven't looked into the patch yet but +1 for the idea.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  3. Re: Column Filtering in Logical Replication

    vignesh C <vignesh21@gmail.com> — 2021-07-01T13:28:58Z

    On Thu, Jul 1, 2021 at 1:06 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
>
> Hi,
>
> Filtering of columns at the publisher node will allow for selective replication of data between publisher and subscriber.  In case the updates on the publisher are targeted only towards specific columns, the user will have an option to reduce network consumption by not sending the data corresponding to new columns that do not change. Note that replica identity values will always be sent irrespective of column filtering settings. The column values that are not sent by the publisher will be populated using local values on the subscriber. For insert command, non-replicated column values will be NULL or the default.
> If column names are not specified while creating or altering a publication,
> all the columns are replicated as per current behaviour.
>
> The proposal for syntax to add table with column names to publication is as follows:
> Create publication:
>
> CREATE PUBLICATION <pub_name> [ FOR TABLE [ONLY] table_name [(colname [,…])] | FOR ALL TABLES]
>
>
> Alter publication:
>
> ALTER PUBLICATION <pub_name> ADD TABLE [ONLY] table_name [(colname [, ..])]
>
>
> Please find attached a patch that implements the above proposal.
> While the patch contains basic implementation and tests, several improvements
> and sanity checks are underway. I will post an updated patch with those changes soon.
>
> Kindly let me know your opinion.

This idea gives more flexibility to the user, +1 for the feature.

Regards,
Vignesh

  4. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-07-06T23:42:51Z

    Hello, here are a few comments on this patch.

The patch adds a function get_att_num_by_name; but we have a lsyscache.c
function for that purpose, get_attnum.  Maybe that one should be used
instead?

get_tuple_columns_map() returns a bitmapset of the attnos of the columns
in the given list, so its name feels wrong.  I propose
get_table_columnset().  However, this function is invoked for every
insert/update change, so it's going to be far too slow to be usable.  I
think you need to cache the bitmapset somewhere, so that the function is
only called on first use.  I didn't look very closely, but it seems that
struct RelationSyncEntry may be a good place to cache it.

The patch adds a new parse node PublicationTable, but doesn't add
copyfuncs.c, equalfuncs.c, readfuncs.c, outfuncs.c support for it.
Maybe try a compile with WRITE_READ_PARSE_PLAN_TREES and/or
COPY_PARSE_PLAN_TREES enabled to make sure everything is covered.
(I didn't verify that this actually catches anything ...)

The new column in pg_publication_rel is prrel_attr.  This name seems at
odds with existing column names (we don't use underscores in catalog
column names).  Maybe prattrs is good enough?  prrelattrs?  We tend to
use plurals for columns that are arrays.

It's not super clear to me that strlist_to_textarray() and related
processing will behave sanely when the column names contain weird
characters such as commas or quotes, or just when used with uppercase
column names.  Maybe it's worth having tests that try to break such
cases.

You seem to have left a debugging "elog(LOG)" line in OpenTableList.

I got warnings from "git am" about trailing whitespace being added by
the patch in two places.


Thanks!

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/

  5. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2021-07-08T03:27:27Z

    Hi, I was wondering if/when a subset of cols is specified then does
that mean it will be possible for the table to be replicated to a
*smaller* table at the subscriber side?

e.g Can a table with 7 cols replicated to a table with 2 cols?

table tab1(a,b,c,d,e,f,g) --> CREATE PUBLICATION pub1 FOR TABLE
tab1(a,b)  --> table tab1(a,b)

~~

I thought maybe that should be possible, but the expected behaviour
for that scenario was not very clear to me from the thread/patch
comments. And the new TAP test uses the tab1 table created exactly the
same for pub/sub, so I couldn't tell from the test code either.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  6. Re: Column Filtering in Logical Replication

    Rahila Syed <rahilasyed90@gmail.com> — 2021-07-12T08:32:26Z

    Hi Peter,

Hi, I was wondering if/when a subset of cols is specified then does
> that mean it will be possible for the table to be replicated to a
> *smaller* table at the subscriber side?
>
e.g Can a table with 7 cols replicated to a table with 2 cols?
>
> table tab1(a,b,c,d,e,f,g) --> CREATE PUBLICATION pub1 FOR TABLE
> tab1(a,b)  --> table tab1(a,b)
>
> ~~


> I thought maybe that should be possible, but the expected behaviour
> for that scenario was not very clear to me from the thread/patch
> comments. And the new TAP test uses the tab1 table created exactly the
> same for pub/sub, so I couldn't tell from the test code either.
>

Currently, this capability is not included in the patch. If the table on
the subscriber
server has lesser attributes than that on the publisher server, it throws
an error at the
time of CREATE SUBSCRIPTION.

About having such a functionality, I don't immediately see any issue with
it as long
as we make sure replica identity columns are always present on both
instances.
However, need to carefully consider situations in which a server subscribes
to multiple
publications,  each publishing a different subset of columns of a table.


Thank you,
Rahila Syed

  7. Re: Column Filtering in Logical Replication

    Rahila Syed <rahilasyed90@gmail.com> — 2021-07-12T09:38:23Z

    Hi Alvaro,

Thank you for comments.

The patch adds a function get_att_num_by_name; but we have a lsyscache.c
> function for that purpose, get_attnum.  Maybe that one should be used
> instead?
>
> Thank you for pointing that out, I agree it makes sense to reuse the
existing function.
Changed it accordingly in the attached patch.


> get_tuple_columns_map() returns a bitmapset of the attnos of the columns
> in the given list, so its name feels wrong.  I propose
> get_table_columnset().  However, this function is invoked for every
> insert/update change, so it's going to be far too slow to be usable.  I
> think you need to cache the bitmapset somewhere, so that the function is
> only called on first use.  I didn't look very closely, but it seems that
> struct RelationSyncEntry may be a good place to cache it.
>
> Makes sense, changed accordingly.


> The patch adds a new parse node PublicationTable, but doesn't add
> copyfuncs.c, equalfuncs.c, readfuncs.c, outfuncs.c support for it.
> Maybe try a compile with WRITE_READ_PARSE_PLAN_TREES and/or
> COPY_PARSE_PLAN_TREES enabled to make sure everything is covered.
> (I didn't verify that this actually catches anything ...)
>

I will test this and include these changes in the next version.


> The new column in pg_publication_rel is prrel_attr.  This name seems at
> odds with existing column names (we don't use underscores in catalog
> column names).  Maybe prattrs is good enough?  prrelattrs?  We tend to
> use plurals for columns that are arrays.
>
> Renamed it to prattrs as per suggestion.

It's not super clear to me that strlist_to_textarray() and related
> processing will behave sanely when the column names contain weird
> characters such as commas or quotes, or just when used with uppercase
> column names.  Maybe it's worth having tests that try to break such
> cases.
>
> Sure, I will include these tests in the next version.


> You seem to have left a debugging "elog(LOG)" line in OpenTableList.
>
> Removed.


> I got warnings from "git am" about trailing whitespace being added by
> the patch in two places.
>
> Should be fixed now.

Thank you,
Rahila Syed

  8. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2021-07-12T12:42:55Z

    
On 7/12/21 10:32 AM, Rahila Syed wrote:
> Hi Peter,
> 
>     Hi, I was wondering if/when a subset of cols is specified then does
>     that mean it will be possible for the table to be replicated to a
>     *smaller* table at the subscriber side? 
> 
>     e.g Can a table with 7 cols replicated to a table with 2 cols?
> 
>     table tab1(a,b,c,d,e,f,g) --> CREATE PUBLICATION pub1 FOR TABLE
>     tab1(a,b)  --> table tab1(a,b)
> 
>     ~~
> 
> 
>     I thought maybe that should be possible, but the expected behaviour
>     for that scenario was not very clear to me from the thread/patch
>     comments. And the new TAP test uses the tab1 table created exactly the
>     same for pub/sub, so I couldn't tell from the test code either.
> 
>  
> Currently, this capability is not included in the patch. If the table on
> the subscriber
> server has lesser attributes than that on the publisher server, it
> throws an error at the 
> time of CREATE SUBSCRIPTION.
> 

That's a bit surprising, to be honest. I do understand the patch simply
treats the filtered columns as "unchanged" because that's the simplest
way to filter the *data* of the columns. But if someone told me we can
"filter columns" I'd expect this to work without the columns on the
subscriber.

> About having such a functionality, I don't immediately see any issue
> with it as long
> as we make sure replica identity columns are always present on both
> instances.

Yeah, that seems like an inherent requirement.

> However, need to carefully consider situations in which a server
> subscribes to multiple 
> publications,  each publishing a different subset of columns of a table.  
>  

Isn't that pretty much the same situation as for multiple subscriptions
each with a different set of I/U/D operations? IIRC we simply merge
those, so why not to do the same thing here and merge the attributes?


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  9. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2021-07-12T12:54:26Z

    
On 7/12/21 11:38 AM, Rahila Syed wrote:
> Hi Alvaro,
> 
> Thank you for comments.
> 
>     The patch adds a function get_att_num_by_name; but we have a lsyscache.c
>     function for that purpose, get_attnum.  Maybe that one should be used
>     instead?
> 
> Thank you for pointing that out, I agree it makes sense to reuse the
> existing function.
> Changed it accordingly in the attached patch.
>  
> 
>     get_tuple_columns_map() returns a bitmapset of the attnos of the columns
>     in the given list, so its name feels wrong.  I propose
>     get_table_columnset().  However, this function is invoked for every
>     insert/update change, so it's going to be far too slow to be usable.  I
>     think you need to cache the bitmapset somewhere, so that the function is
>     only called on first use.  I didn't look very closely, but it seems that
>     struct RelationSyncEntry may be a good place to cache it.
> 
> Makes sense, changed accordingly.
>  

To nitpick, I find "Bitmapset *att_list" a bit annoying, because it's
not really a list ;-)


FWIW "make check" fails for me with this version, due to segfault in
OpenTableLists. Apparenly there's some confusion - the code expects the
list to contain PublicationTable nodes, and tries to extract the
RangeVar from the elements. But the list actually contains RangeVar, so
this crashes and burns. See the attached backtrace.

I'd bet this is because the patch uses list of RangeVar in some cases
and list of PublicationTable in some cases, similarly to the "row
filtering" patch nearby. IMHO this is just confusing and we should
always pass list of PublicationTable nodes.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  10. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-07-12T14:53:01Z

    On 2021-Jul-12, Tomas Vondra wrote:

> FWIW "make check" fails for me with this version, due to segfault in
> OpenTableLists. Apparenly there's some confusion - the code expects the
> list to contain PublicationTable nodes, and tries to extract the
> RangeVar from the elements. But the list actually contains RangeVar, so
> this crashes and burns. See the attached backtrace.
> 
> I'd bet this is because the patch uses list of RangeVar in some cases
> and list of PublicationTable in some cases, similarly to the "row
> filtering" patch nearby. IMHO this is just confusing and we should
> always pass list of PublicationTable nodes.

+1 don't make the code guess what type of list it is.  Changing all the
uses of that node to deal with PublicationTable seems best.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Cuando no hay humildad las personas se degradan" (A. Christie)

  11. Re: Column Filtering in Logical Replication

    Rahila Syed <rahilasyed90@gmail.com> — 2021-07-13T14:43:44Z

    Hi Tomas,

Thank you for your comments.


>
> >
> > Currently, this capability is not included in the patch. If the table on
> > the subscriber
> > server has lesser attributes than that on the publisher server, it
> > throws an error at the
> > time of CREATE SUBSCRIPTION.
> >
>
> That's a bit surprising, to be honest. I do understand the patch simply
> treats the filtered columns as "unchanged" because that's the simplest
> way to filter the *data* of the columns. But if someone told me we can
> "filter columns" I'd expect this to work without the columns on the
> subscriber.
>
> OK, I will look into adding this.


>
> > However, need to carefully consider situations in which a server
> > subscribes to multiple
> > publications,  each publishing a different subset of columns of a table.

Isn't that pretty much the same situation as for multiple subscriptions
> each with a different set of I/U/D operations? IIRC we simply merge
> those, so why not to do the same thing here and merge the attributes?
>
>
Yeah, I agree with the solution to merge the attributes, similar to how
operations are merged. My concern was also from an implementation point
of view, will it be a very drastic change. I now had a look at how remote
relation
attributes are acquired for comparison with local attributes at the
subscriber.
It seems that the publisher will need to send the information about the
filtered columns
for each publication specified during CREATE SUBSCRIPTION.
This will be read at the subscriber side which in turn updates its cache
accordingly.
Currently, the subscriber expects all attributes of a published relation to
be present.
I will add code for this in the next version of the patch.

 To nitpick, I find "Bitmapset *att_list" a bit annoying, because it's

not really a list ;-)


I will make this change with the next version



>  FWIW "make check" fails for me with this version, due to segfault in
> OpenTableLists. Apparenly there's some confusion - the code expects the
> list to contain PublicationTable nodes, and tries to extract the
> RangeVar from the elements. But the list actually contains RangeVar, so
> this crashes and burns. See the attached backtrace.
>
>
Thank you for the report, This is fixed in the attached version, now all
publication
function calls accept the PublicationTableInfo list.

Thank you,
Rahila Syed

  12. Re: Column Filtering in Logical Replication

    Ibrar Ahmed <ibrar.ahmad@gmail.com> — 2021-07-19T10:20:32Z

    On Tue, Jul 13, 2021 at 7:44 PM Rahila Syed <rahilasyed90@gmail.com> wrote:

>
> Hi Tomas,
>
> Thank you for your comments.
>
>
>>
>> >
>> > Currently, this capability is not included in the patch. If the table on
>> > the subscriber
>> > server has lesser attributes than that on the publisher server, it
>> > throws an error at the
>> > time of CREATE SUBSCRIPTION.
>> >
>>
>> That's a bit surprising, to be honest. I do understand the patch simply
>> treats the filtered columns as "unchanged" because that's the simplest
>> way to filter the *data* of the columns. But if someone told me we can
>> "filter columns" I'd expect this to work without the columns on the
>> subscriber.
>>
>> OK, I will look into adding this.
>
>
>>
>> > However, need to carefully consider situations in which a server
>> > subscribes to multiple
>> > publications,  each publishing a different subset of columns of a
>> table.
>
> Isn't that pretty much the same situation as for multiple subscriptions
>> each with a different set of I/U/D operations? IIRC we simply merge
>> those, so why not to do the same thing here and merge the attributes?
>>
>>
> Yeah, I agree with the solution to merge the attributes, similar to how
> operations are merged. My concern was also from an implementation point
> of view, will it be a very drastic change. I now had a look at how remote
> relation
> attributes are acquired for comparison with local attributes at the
> subscriber.
> It seems that the publisher will need to send the information about the
> filtered columns
> for each publication specified during CREATE SUBSCRIPTION.
> This will be read at the subscriber side which in turn updates its cache
> accordingly.
> Currently, the subscriber expects all attributes of a published relation
> to be present.
> I will add code for this in the next version of the patch.
>
>  To nitpick, I find "Bitmapset *att_list" a bit annoying, because it's
>
> not really a list ;-)
>
>
> I will make this change with the next version
>
>
>
>>  FWIW "make check" fails for me with this version, due to segfault in
>> OpenTableLists. Apparenly there's some confusion - the code expects the
>> list to contain PublicationTable nodes, and tries to extract the
>> RangeVar from the elements. But the list actually contains RangeVar, so
>> this crashes and burns. See the attached backtrace.
>>
>>
> Thank you for the report, This is fixed in the attached version, now all
> publication
> function calls accept the PublicationTableInfo list.
>
> Thank you,
> Rahila Syed
>
>
>

The patch does not apply, and an rebase is required

Hunk #8 succeeded at 1259 (offset 99 lines).
Hunk #9 succeeded at 1360 (offset 99 lines).
1 out of 9 hunks FAILED -- saving rejects to file
src/backend/replication/pgoutput/pgoutput.c.rej
patching file src/include/catalog/pg_publication.h


Changing the status to "Waiting on Author"

-- 
Ibrar Ahmed

  13. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-07-21T00:14:17Z

    Hello,

I think this looks good regarding the PublicationRelationInfo API that was
discussed.

Looking at OpenTableList(), I think you forgot to update the comment --
it says "open relations specified by a RangeVar list", but the list is
now of PublicationTable.  Also I think it would be good to say that the
returned tables are PublicationRelationInfo, maybe such as "In the
returned list of PublicationRelationInfo, the tables are locked ..."

In AlterPublicationTables() I was confused by some code that seemed
commented a bit too verbosely (for a moment I thought the whole list was
being copied into a different format).  May I suggest something more
compact like

			/* Not yet in list; open it and add it to the list */
			if (!found)
			{
				Relation	oldrel;
				PublicationRelationInfo *pubrel;
			   
				oldrel = table_open(oldrelid, ShareUpdateExclusiveLock);

				/* Wrap it in PublicationRelationInfo */
				pubrel = palloc(sizeof(PublicationRelationInfo));
				pubrel->relation = oldrel;
				pubrel->relid = oldrelid;
				pubrel->columns = NIL;		/* not needed */

				delrels = lappend(delrels, pubrel);
			}

Thanks!

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/

  14. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-07-22T18:48:32Z

    One thing I just happened to notice is this part of your commit message

: REPLICA IDENTITY columns are always replicated
: irrespective of column names specification.

... for which you don't have any tests -- I mean, create a table with a
certain REPLICA IDENTITY and later try to publish a set of columns that
doesn't include all the columns in the replica identity, then verify
that those columns are indeed published.

Having said that, I'm not sure I agree with this design decision; what I
think this is doing is hiding from the user the fact that they are
publishing columns that they don't want to publish.  I think as a user I
would rather get an error in that case:

  ERROR:  invalid column list in published set
  DETAIL:  The set of published commands does not include all the replica identity columns.

or something like that.  Avoid possible nasty surprises of security-
leaking nature.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"On the other flipper, one wrong move and we're Fatal Exceptions"
(T.U.X.: Term Unit X  - http://www.thelinuxreview.com/TUX/)

  15. Re: Column Filtering in Logical Replication

    Rahila Syed <rahilasyed90@gmail.com> — 2021-08-08T19:59:50Z

    Hi,



> >
>> > Currently, this capability is not included in the patch. If the table on
>> > the subscriber
>> > server has lesser attributes than that on the publisher server, it
>> > throws an error at the
>> > time of CREATE SUBSCRIPTION.
>> >
>>
>> That's a bit surprising, to be honest. I do understand the patch simply
>> treats the filtered columns as "unchanged" because that's the simplest
>> way to filter the *data* of the columns. But if someone told me we can
>> "filter columns" I'd expect this to work without the columns on the
>> subscriber.
>>
>> OK, I will look into adding this.
>

This has been added in the attached patch. Now, instead of
treating the filtered columns as unchanged and sending a byte
with that information, unfiltered columns are not sent to the subscriber
server at all. This along with saving the network bandwidth, allows
the logical replication to even work between tables with different numbers
of
columns i.e with the table on subscriber server containing only the
filtered
columns. Currently, replica identity columns are replicated irrespective of
the presence of the column filters, hence the table on the subscriber side
must
contain the replica identity columns.

The patch adds a new parse node PublicationTable, but doesn't add
> copyfuncs.c, equalfuncs.c, readfuncs.c, outfuncs.c support for it.


Thanks, added this.


> Looking at OpenTableList(), I think you forgot to update the comment --
> it says "open relations specified by a RangeVar list",


Thank you for the review, Modified this.

To nitpick, I find "Bitmapset *att_list" a bit annoying, because it's
> not really a list ;-)


Changed this.

>
>  It's not super clear to me that strlist_to_textarray() and related
> processing will behave sanely when the column names contain weird
> characters such as commas or quotes, or just when used with uppercase
> column names.  Maybe it's worth having tests that try to break such
> cases.


Added a few test cases for this.

In AlterPublicationTables() I was confused by some code that seemed
> commented a bit too verbosely


Modified this as per the suggestion.

: REPLICA IDENTITY columns are always replicated
> : irrespective of column names specification.


... for which you don't have any tests


I have added these tests.

Having said that, I'm not sure I agree with this design decision; what I
> think this is doing is hiding from the user the fact that they are
> publishing columns that they don't want to publish.  I think as a user I
> would rather get an error in that case:


  ERROR:  invalid column list in published set
>   DETAIL:  The set of published commands does not include all the replica
> identity columns.


or something like that.  Avoid possible nasty surprises of security-
> leaking nature.


Ok, Thank you for your opinion. I agree that giving an explicit error in
this case will be safer.
I will include this, in case there are no counter views.

Thank you for your review comments. Please find attached the rebased and
updated patch.


Thank you,
Rahila Syed

  16. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-08-09T10:29:16Z

    On Mon, Aug 9, 2021 at 1:36 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
>
>> Having said that, I'm not sure I agree with this design decision; what I
>> think this is doing is hiding from the user the fact that they are
>> publishing columns that they don't want to publish.  I think as a user I
>> would rather get an error in that case:
>
>
>>   ERROR:  invalid column list in published set
>>   DETAIL:  The set of published commands does not include all the replica identity columns.
>
>
>> or something like that.  Avoid possible nasty surprises of security-
>> leaking nature.
>
>
> Ok, Thank you for your opinion. I agree that giving an explicit error in this case will be safer.
>

+1 for an explicit error in this case.

Can you please explain why you have the restriction for including
replica identity columns and do we want to put a similar restriction
for the primary key? As far as I understand, if we allow default
values on subscribers for replica identity, then probably updates,
deletes won't work as they need to use replica identity (or PK) to
search the required tuple. If so, shouldn't we add this restriction
only when a publication has been defined for one of these (Update,
Delete) actions?

Another point is what if someone drops the column used in one of the
publications? Do we want to drop the entire relation from publication
or just remove the column filter or something else?

Do we want to consider that the columns specified in the filter must
not have NOT NULL constraint? Because, otherwise, the subscriber will
error out inserting such rows?

Minor comments:
================
  pq_sendbyte(out, flags);
-
  /* attribute name */
  pq_sendstring(out, NameStr(att->attname));

@@ -953,6 +1000,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel)

  /* attribute mode */
  pq_sendint32(out, att->atttypmod);
+
  }

  bms_free(idattrs);
diff --git a/src/backend/replication/logical/relation.c
b/src/backend/replication/logical/relation.c
index c37e2a7e29..d7a7b00841 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -354,7 +354,6 @@ logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode)

  attnum = logicalrep_rel_att_by_name(remoterel,
  NameStr(attr->attname));
-
  entry->attrmap->attnums[i] = attnum;

There are quite a few places in the patch that contains spurious line
additions or removals.

-- 
With Regards,
Amit Kapila.

  17. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-08-09T10:45:32Z

    On Mon, Aug 9, 2021 at 3:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Aug 9, 2021 at 1:36 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
> >
> >> Having said that, I'm not sure I agree with this design decision; what I
> >> think this is doing is hiding from the user the fact that they are
> >> publishing columns that they don't want to publish.  I think as a user I
> >> would rather get an error in that case:
> >
> >
> >>   ERROR:  invalid column list in published set
> >>   DETAIL:  The set of published commands does not include all the replica identity columns.
> >
> >
> >> or something like that.  Avoid possible nasty surprises of security-
> >> leaking nature.
> >
> >
> > Ok, Thank you for your opinion. I agree that giving an explicit error in this case will be safer.
> >
>
> +1 for an explicit error in this case.
>
> Can you please explain why you have the restriction for including
> replica identity columns and do we want to put a similar restriction
> for the primary key? As far as I understand, if we allow default
> values on subscribers for replica identity, then probably updates,
> deletes won't work as they need to use replica identity (or PK) to
> search the required tuple. If so, shouldn't we add this restriction
> only when a publication has been defined for one of these (Update,
> Delete) actions?
>
> Another point is what if someone drops the column used in one of the
> publications? Do we want to drop the entire relation from publication
> or just remove the column filter or something else?
>
> Do we want to consider that the columns specified in the filter must
> not have NOT NULL constraint? Because, otherwise, the subscriber will
> error out inserting such rows?
>

I noticed that other databases provide this feature [1] and they allow
users to specify "Columns that are included in Filter" or specify "All
columns to be included in filter except for a subset of columns". I am
not sure if want to provide both ways in the first version but at
least we should consider it as a future extensibility requirement and
try to choose syntax accordingly.

[1] - https://docs.oracle.com/en/cloud/paas/goldengate-cloud/gwuad/selecting-columns.html#GUID-9A851C8B-48F7-43DF-8D98-D086BE069E20

-- 
With Regards,
Amit Kapila.

  18. Re: Column Filtering in Logical Replication

    Rahila Syed <rahilasyed90@gmail.com> — 2021-08-12T03:10:00Z

    Hi Amit,

Thanks for your review.


> Can you please explain why you have the restriction for including
> replica identity columns and do we want to put a similar restriction
> for the primary key? As far as I understand, if we allow default
> values on subscribers for replica identity, then probably updates,
> deletes won't work as they need to use replica identity (or PK) to
> search the required tuple. If so, shouldn't we add this restriction
> only when a publication has been defined for one of these (Update,
> Delete) actions?
>

Yes, like you mentioned they are needed for Updates and Deletes to work.
The restriction for including replica identity columns in column filters
exists because
In case the replica identity column values did not change, the old row
replica identity columns
are not sent to the subscriber, thus we would need new replica identity
columns
to be sent to identify the row that is to be Updated or Deleted.
I haven't tested if it would break Insert as well  though. I will update
the patch accordingly.


> Another point is what if someone drops the column used in one of the
> publications? Do we want to drop the entire relation from publication
> or just remove the column filter or something else?
>
>
Thanks for pointing this out. Currently, this is not handled in the patch.
I think dropping the column from the filter would make sense on the lines
of the table being dropped from publication, in case of drop table.


> Do we want to consider that the columns specified in the filter must
> not have NOT NULL constraint? Because, otherwise, the subscriber will
> error out inserting such rows?
>
> I think you mean columns *not* specified in the filter must not have NOT
NULL constraint
on the subscriber, as this will break during insert, as it will try to
insert NULL for columns
not sent by the publisher.
I will look into fixing this. Probably this won't be a problem in
case the column is auto generated or contains a default value.


> Minor comments:
> ================
>   pq_sendbyte(out, flags);
> -
>   /* attribute name */
>   pq_sendstring(out, NameStr(att->attname));
>
> @@ -953,6 +1000,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
>
>   /* attribute mode */
>   pq_sendint32(out, att->atttypmod);
> +
>   }
>
>   bms_free(idattrs);
> diff --git a/src/backend/replication/logical/relation.c
> b/src/backend/replication/logical/relation.c
> index c37e2a7e29..d7a7b00841 100644
> --- a/src/backend/replication/logical/relation.c
> +++ b/src/backend/replication/logical/relation.c
> @@ -354,7 +354,6 @@ logicalrep_rel_open(LogicalRepRelId remoteid,
> LOCKMODE lockmode)
>
>   attnum = logicalrep_rel_att_by_name(remoterel,
>   NameStr(attr->attname));
> -
>   entry->attrmap->attnums[i] = attnum;
>
> There are quite a few places in the patch that contains spurious line
> additions or removals.
>
>
Thank you for your comments, I will fix these.

Thank you,
Rahila Syed

  19. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-08-12T08:29:27Z

    On Thu, Aug 12, 2021 at 8:40 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
>
>>
>> Can you please explain why you have the restriction for including
>> replica identity columns and do we want to put a similar restriction
>> for the primary key? As far as I understand, if we allow default
>> values on subscribers for replica identity, then probably updates,
>> deletes won't work as they need to use replica identity (or PK) to
>> search the required tuple. If so, shouldn't we add this restriction
>> only when a publication has been defined for one of these (Update,
>> Delete) actions?
>
>
> Yes, like you mentioned they are needed for Updates and Deletes to work.
> The restriction for including replica identity columns in column filters exists because
> In case the replica identity column values did not change, the old row replica identity columns
> are not sent to the subscriber, thus we would need new replica identity columns
> to be sent to identify the row that is to be Updated or Deleted.
> I haven't tested if it would break Insert as well  though. I will update the patch accordingly.
>

Okay, but then we also need to ensure that the user shouldn't be
allowed to enable the 'update' or 'delete' for a publication that
contains some filter that doesn't have replica identity columns.

>>
>> Another point is what if someone drops the column used in one of the
>> publications? Do we want to drop the entire relation from publication
>> or just remove the column filter or something else?
>>
>
> Thanks for pointing this out. Currently, this is not handled in the patch.
> I think dropping the column from the filter would make sense on the lines
> of the table being dropped from publication, in case of drop table.
>

I think it would be tricky if you want to remove the column from the
filter because you need to recompute the entire filter and update it
again. Also, you might need to do this for all the publications that
have a particular column in their filter clause. It might be easier to
drop the entire filter but you can check if it is easier another way
than it is good.

>>
>> Do we want to consider that the columns specified in the filter must
>> not have NOT NULL constraint? Because, otherwise, the subscriber will
>> error out inserting such rows?
>>
> I think you mean columns *not* specified in the filter must not have NOT NULL constraint
> on the subscriber, as this will break during insert, as it will try to insert NULL for columns
> not sent by the publisher.
>

Right.

-- 
With Regards,
Amit Kapila.

  20. Re: Column Filtering in Logical Replication

    Rahila Syed <rahilasyed90@gmail.com> — 2021-09-01T21:21:41Z

    Hi,


>> Another point is what if someone drops the column used in one of the
>> publications? Do we want to drop the entire relation from publication
>> or just remove the column filter or something else?
>>
>
After thinking about this, I think it is best to remove the entire table
from publication,
if a column specified in the column filter is dropped from the table.
Because, if we drop the entire filter without dropping the table, it means
all the columns will be replicated,
and the downstream server table might not have those columns.
If we drop only the column from the filter we might have to recreate the
filter and check for replica identity.
That means if the replica identity column is dropped, you can't drop it
from the filter,
and might have to drop the entire publication-table mapping anyways.

Thus, I think it is cleanest to drop the entire relation from publication.

This has been implemented in the attached version.


> Do we want to consider that the columns specified in the filter must
>> not have NOT NULL constraint? Because, otherwise, the subscriber will
>> error out inserting such rows?
>>
>> I think you mean columns *not* specified in the filter must not have NOT
> NULL constraint
> on the subscriber, as this will break during insert, as it will try to
> insert NULL for columns
> not sent by the publisher.
> I will look into fixing this. Probably this won't be a problem in
> case the column is auto generated or contains a default value.
>
>
I am not sure if this needs to be handled. Ideally, we need to prevent the
subscriber tables from having a NOT NULL
constraint if the publisher uses column filters to publish the values of
the table. There is no way
to do this at the time of creating a table on subscriber.
As this involves querying the publisher for this information, it can be
done at the time of initial table synchronization.
i.e error out if any of the subscribed tables has NOT NULL constraint on
non-filter columns.
This will lead to the user dropping and recreating the subscription after
removing the
NOT NULL constraint from the table.
I think the same can be achieved by doing nothing and letting the
subscriber error out while inserting rows.

Minor comments:
>> ================
>>   pq_sendbyte(out, flags);
>> -
>>   /* attribute name */
>>   pq_sendstring(out, NameStr(att->attname));
>>
>> @@ -953,6 +1000,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
>>
>>   /* attribute mode */
>>   pq_sendint32(out, att->atttypmod);
>> +
>>   }
>>
>>   bms_free(idattrs);
>> diff --git a/src/backend/replication/logical/relation.c
>> b/src/backend/replication/logical/relation.c
>> index c37e2a7e29..d7a7b00841 100644
>> --- a/src/backend/replication/logical/relation.c
>> +++ b/src/backend/replication/logical/relation.c
>> @@ -354,7 +354,6 @@ logicalrep_rel_open(LogicalRepRelId remoteid,
>> LOCKMODE lockmode)
>>
>>   attnum = logicalrep_rel_att_by_name(remoterel,
>>   NameStr(attr->attname));
>> -
>>   entry->attrmap->attnums[i] = attnum;
>>
>> There are quite a few places in the patch that contains spurious line
>> additions or removals.
>>
>>
> Fixed these in the attached patch.

Having said that, I'm not sure I agree with this design decision; what I
> think this is doing is hiding from the user the fact that they are
> publishing columns that they don't want to publish.  I think as a user I
> would rather get an error in that case:


  ERROR:  invalid column list in published set
>   DETAIL:  The set of published commands does not include all the replica
> identity columns.


Added this.

Also added some more tests. Please find attached a rebased and updated
patch.

Thank you,
Rahila Syed

  21. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2021-09-02T07:53:59Z

    On Thu, Sep 2, 2021 at 7:21 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
>
...
>
> Also added some more tests. Please find attached a rebased and updated patch.

I fetched and applied the v4 patch.

It applied cleanly, and the build and make check was OK.

But I encountered some errors running the TAP subscription tests, as follows:

...
t/018_stream_subxact_abort.pl ...... ok
t/019_stream_subxact_ddl_abort.pl .. ok
t/020_messages.pl .................. ok
t/021_column_filter.pl ............. 1/9
#   Failed test 'insert on column c is not replicated'
#   at t/021_column_filter.pl line 126.
#          got: ''
#     expected: '1|abc'

#   Failed test 'update on column c is not replicated'
#   at t/021_column_filter.pl line 130.
#          got: ''
#     expected: '1|abc'
# Looks like you failed 2 tests of 9.
t/021_column_filter.pl ............. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/9 subtests
t/021_twophase.pl .................. ok
t/022_twophase_cascade.pl .......... ok
t/023_twophase_stream.pl ........... ok
t/024_add_drop_pub.pl .............. ok
t/100_bugs.pl ...................... ok

Test Summary Report
-------------------
t/021_column_filter.pl           (Wstat: 512 Tests: 9 Failed: 2)
  Failed tests:  6-7
  Non-zero exit status: 2
Files=26, Tests=263, 192 wallclock secs ( 0.57 usr  0.09 sys + 110.17
cusr 25.45 csys = 136.28 CPU)
Result: FAIL
make: *** [check] Error 1

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  22. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-02T08:49:08Z

    On 2021-Sep-02, Rahila Syed wrote:

> After thinking about this, I think it is best to remove the entire table
> from publication,
> if a column specified in the column filter is dropped from the table.

Hmm, I think it would be cleanest to give responsibility to the user: if
the column to be dropped is in the filter, then raise an error, aborting
the drop.  Then it is up to them to figure out what to do.




-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
"El destino baraja y nosotros jugamos" (A. Schopenhauer)

  23. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-02T08:56:39Z

    I think the WITH RECURSIVE query would be easier and more performant by
using pg_partition_tree and pg_partition_root.


-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
"Porque Kim no hacía nada, pero, eso sí,
con extraordinario éxito" ("Kim", Kipling)

  24. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-04T04:11:30Z

    On Thu, Sep 2, 2021 at 2:19 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-02, Rahila Syed wrote:
>
> > After thinking about this, I think it is best to remove the entire table
> > from publication,
> > if a column specified in the column filter is dropped from the table.
>
> Hmm, I think it would be cleanest to give responsibility to the user: if
> the column to be dropped is in the filter, then raise an error, aborting
> the drop.
>

Do you think that will make sense if the user used Cascade (Alter
Table ... Drop Column ... Cascade)?

-- 
With Regards,
Amit Kapila.

  25. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-04T04:42:53Z

    On Thu, Sep 2, 2021 at 2:51 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
>
>>>
>>> Do we want to consider that the columns specified in the filter must
>>> not have NOT NULL constraint? Because, otherwise, the subscriber will
>>> error out inserting such rows?
>>>
>> I think you mean columns *not* specified in the filter must not have NOT NULL constraint
>> on the subscriber, as this will break during insert, as it will try to insert NULL for columns
>> not sent by the publisher.
>> I will look into fixing this. Probably this won't be a problem in
>> case the column is auto generated or contains a default value.
>>
>
> I am not sure if this needs to be handled. Ideally, we need to prevent the subscriber tables from having a NOT NULL
> constraint if the publisher uses column filters to publish the values of the table. There is no way
> to do this at the time of creating a table on subscriber.
>
> As this involves querying the publisher for this information, it can be done at the time of initial table synchronization.
> i.e error out if any of the subscribed tables has NOT NULL constraint on non-filter columns.
> This will lead to the user dropping and recreating the subscription after removing the
> NOT NULL constraint from the table.
> I think the same can be achieved by doing nothing and letting the subscriber error out while inserting rows.
>

That makes sense and also it is quite possible that users don't have
such columns in the tables on subscribers. I guess we can add such a
recommendation in the docs instead of doing anything in the code.

Few comments:
============
1.
+
+ /*
+ * Cannot specify column filter when REPLICA IDENTITY IS FULL
+ * or if column filter does not contain REPLICA IDENITY columns
+ */
+ if (targetcols != NIL)
+ {
+ if (replidentfull)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter with REPLICA IDENTITY FULL")));

Why do we want to have such a restriction for REPLICA IDENTITY FULL? I
think it is better to expand comments in that regards.

2.
@@ -839,7 +839,6 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
  ereport(ERROR,
  (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
  errmsg("extra data after last expected column")));
-
  fieldno = 0;

@@ -944,7 +992,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
  flags |= LOGICALREP_IS_REPLICA_IDENTITY;

  pq_sendbyte(out, flags);
-
  /* attribute name */
  pq_sendstring(out, NameStr(att->attname));

@@ -953,6 +1000,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel)

  /* attribute mode */
  pq_sendint32(out, att->atttypmod);
+
  }

Spurious line removals and addition.

-- 
With Regards,
Amit Kapila.

  26. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-04T05:00:17Z

    On Sat, Sep 4, 2021 at 10:12 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Sep 2, 2021 at 2:51 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
> >
> >>>
> >>> Do we want to consider that the columns specified in the filter must
> >>> not have NOT NULL constraint? Because, otherwise, the subscriber will
> >>> error out inserting such rows?
> >>>
> >> I think you mean columns *not* specified in the filter must not have NOT NULL constraint
> >> on the subscriber, as this will break during insert, as it will try to insert NULL for columns
> >> not sent by the publisher.
> >> I will look into fixing this. Probably this won't be a problem in
> >> case the column is auto generated or contains a default value.
> >>
> >
> > I am not sure if this needs to be handled. Ideally, we need to prevent the subscriber tables from having a NOT NULL
> > constraint if the publisher uses column filters to publish the values of the table. There is no way
> > to do this at the time of creating a table on subscriber.
> >
> > As this involves querying the publisher for this information, it can be done at the time of initial table synchronization.
> > i.e error out if any of the subscribed tables has NOT NULL constraint on non-filter columns.
> > This will lead to the user dropping and recreating the subscription after removing the
> > NOT NULL constraint from the table.
> > I think the same can be achieved by doing nothing and letting the subscriber error out while inserting rows.
> >
>
> That makes sense and also it is quite possible that users don't have
> such columns in the tables on subscribers. I guess we can add such a
> recommendation in the docs instead of doing anything in the code.
>
> Few comments:
> ============
>

Did you give any thoughts to my earlier suggestion related to syntax [1]?

[1] - https://www.postgresql.org/message-id/CAA4eK1J9b_0_PMnJ2jq9E55bcbmTKdUmy6jPnkf1Zwy2jxah_g%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  27. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-04T14:41:04Z

    On 2021-Sep-04, Amit Kapila wrote:

> On Thu, Sep 2, 2021 at 2:19 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> >
> > On 2021-Sep-02, Rahila Syed wrote:
> >
> > > After thinking about this, I think it is best to remove the entire table
> > > from publication,
> > > if a column specified in the column filter is dropped from the table.
> >
> > Hmm, I think it would be cleanest to give responsibility to the user: if
> > the column to be dropped is in the filter, then raise an error, aborting
> > the drop.
> 
> Do you think that will make sense if the user used Cascade (Alter
> Table ... Drop Column ... Cascade)?

... ugh.  Since CASCADE is already defined to be a potentially-data-loss
operation, then that may be acceptable behavior.  For sure the default
RESTRICT behavior shouldn't do it, though.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/

  28. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-06T03:23:39Z

    On Sat, Sep 4, 2021 at 8:11 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-04, Amit Kapila wrote:
>
> > On Thu, Sep 2, 2021 at 2:19 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> > >
> > > On 2021-Sep-02, Rahila Syed wrote:
> > >
> > > > After thinking about this, I think it is best to remove the entire table
> > > > from publication,
> > > > if a column specified in the column filter is dropped from the table.
> > >
> > > Hmm, I think it would be cleanest to give responsibility to the user: if
> > > the column to be dropped is in the filter, then raise an error, aborting
> > > the drop.
> >
> > Do you think that will make sense if the user used Cascade (Alter
> > Table ... Drop Column ... Cascade)?
>
> ... ugh.  Since CASCADE is already defined to be a potentially-data-loss
> operation, then that may be acceptable behavior.  For sure the default
> RESTRICT behavior shouldn't do it, though.
>

That makes sense to me.

-- 
With Regards,
Amit Kapila.

  29. Re: Column Filtering in Logical Replication

    Rahila Syed <rahilasyed90@gmail.com> — 2021-09-06T15:55:42Z

    Hi,

On Mon, Sep 6, 2021 at 8:53 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

> On Sat, Sep 4, 2021 at 8:11 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
> wrote:
> >
> > On 2021-Sep-04, Amit Kapila wrote:
> >
> > > On Thu, Sep 2, 2021 at 2:19 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
> wrote:
> > > >
> > > > On 2021-Sep-02, Rahila Syed wrote:
> > > >
> > > > > After thinking about this, I think it is best to remove the entire
> table
> > > > > from publication,
> > > > > if a column specified in the column filter is dropped from the
> table.
> > > >
> > > > Hmm, I think it would be cleanest to give responsibility to the
> user: if
> > > > the column to be dropped is in the filter, then raise an error,
> aborting
> > > > the drop.
> > >
> > > Do you think that will make sense if the user used Cascade (Alter
> > > Table ... Drop Column ... Cascade)?
> >
> > ... ugh.  Since CASCADE is already defined to be a potentially-data-loss
> > operation, then that may be acceptable behavior.  For sure the default
> > RESTRICT behavior shouldn't do it, though.
> >
>
> That makes sense to me.
>
> However, the default (RESTRICT) behaviour of DROP TABLE allows
removing the table from the publication. I have implemented the removal of
table from publication
on drop column (RESTRICT)  on the same lines.

Although it does make sense to not allow dropping tables from publication,
in case of RESTRICT.
It makes me wonder how DROP TABLE (RESTRICT) allows cascading the drop
table to publication.

Did you give any thoughts to my earlier suggestion related to syntax [1]?


> [1] -
> https://www.postgresql.org/message-id/CAA4eK1J9b_0_PMnJ2jq9E55bcbmTKdUmy6jPnkf1Zwy2jxah_g%40mail.gmail.com


For future support to replicate all columns except (x,y,z), I think some
optional keywords like
COLUMNS NOT IN can be inserted between table name and (*columns_list*) as
follows.
ALTER PUBLICATION ADD TABLE tab_name [COLUMNS NOT IN] (x,y,z)
I think this should be possible as a future addition to proposed syntax in
the patch.
Please let me know your opinion.

Thank you,
Rahila Syed

  30. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-06T17:51:52Z

    On 2021-Sep-06, Rahila Syed wrote:

> > > ... ugh.  Since CASCADE is already defined to be a
> > > potentially-data-loss operation, then that may be acceptable
> > > behavior.  For sure the default RESTRICT behavior shouldn't do it,
> > > though.
> >
> > That makes sense to me.
>
> However, the default (RESTRICT) behaviour of DROP TABLE allows
> removing the table from the publication. I have implemented the
> removal of table from publication on drop column (RESTRICT)  on the
> same lines.

But dropping the table is quite a different action from dropping a
column, isn't it?  If you drop a table, it seems perfectly reasonable
that it has to be removed from the publication -- essentially, when the
user drops a table, she is saying "I don't care about this table
anymore".  However, if you drop just one column, that doesn't
necessarily mean that the user wants to stop publishing the whole table.
Removing the table from the publication in ALTER TABLE DROP COLUMN seems
like an overreaction.  (Except perhaps in the special case were the
column being dropped is the only one that was being published.)

So let's discuss what should happen.  If you drop a column, and the
column is filtered out, then it seems to me that the publication should
continue to have the table, and it should continue to filter out the
other columns that were being filtered out, regardless of CASCADE/RESTRICT.
However, if the column is *included* in the publication, and you drop
it, ISTM there are two cases:

1. If it's DROP CASCADE, then the list of columns to replicate should
continue to have all columns it previously had, so just remove the
column that is being dropped.

2. If it's DROP RESTRICT, then an error should be raised so that the
user can make a concious decision to remove the column from the filter
before dropping the column.

> Did you give any thoughts to my earlier suggestion related to syntax [1]?
> 
> [1] https://www.postgresql.org/message-id/CAA4eK1J9b_0_PMnJ2jq9E55bcbmTKdUmy6jPnkf1Zwy2jxah_g%40mail.gmail.com

This is a great followup idea, after the current feature is committed.
There are a few things that have been reported in review comments; let's
get those addressed before adding more features on top.

I pushed the clerical part of this -- namely the addition of
PublicationTable node and PublicationRelInfo struct.  I attach the part
of your v4 patch that I didn't include.  It contains a couple of small
corrections, but I didn't do anything invasive (such as pgindent)
because that would perhaps cause you too much merge pain.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/

  31. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-06T19:16:03Z

    The code in get_rel_sync_entry() changes current memory context to
CacheMemoryContext, then does a bunch of memory-leaking things.  This is
not good, because that memory context has to be very carefully managed
to avoid permanent memory leaks.  I suppose you added that because you
need something -- probably entry->att_map -- to survive memory context
resets, but if so then you need to change to CacheMemoryContext only
when that memory is allocated, not other chunks of memory.  I suspect
you can fix this by moving the MemoryContextSwitchTo() to just before
calling get_table_columnset; then all the leaky thinkgs are done in
whatever the original memory context is, which is fine.

(However, you also need to make sure that ->att_map is carefully freed
at the right time.  It looks like this already happens in
rel_sync_cache_relation_cb, but is rel_sync_cache_publication_cb
correct?  And in get_rel_sync_entry() itself, what if the entry already
has att_map -- should it be freed prior to allocating another one?)

By the way, I notice that your patch doesn't add documentation changes,
which are of course necessary.



/me is left wondering about PGOutputData->publication_names memory
handling ...

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/

  32. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2021-09-06T21:03:50Z

    
On 9/6/21 7:51 PM, Alvaro Herrera wrote:
> On 2021-Sep-06, Rahila Syed wrote:
> 
>>>> ... ugh.  Since CASCADE is already defined to be a
>>>> potentially-data-loss operation, then that may be acceptable
>>>> behavior.  For sure the default RESTRICT behavior shouldn't do it,
>>>> though.
>>>
>>> That makes sense to me.
>>
>> However, the default (RESTRICT) behaviour of DROP TABLE allows
>> removing the table from the publication. I have implemented the
>> removal of table from publication on drop column (RESTRICT)  on the
>> same lines.
> 
> But dropping the table is quite a different action from dropping a
> column, isn't it?  If you drop a table, it seems perfectly reasonable
> that it has to be removed from the publication -- essentially, when the
> user drops a table, she is saying "I don't care about this table
> anymore".  However, if you drop just one column, that doesn't
> necessarily mean that the user wants to stop publishing the whole table.
> Removing the table from the publication in ALTER TABLE DROP COLUMN seems
> like an overreaction.  (Except perhaps in the special case were the
> column being dropped is the only one that was being published.)
> 
> So let's discuss what should happen.  If you drop a column, and the
> column is filtered out, then it seems to me that the publication should
> continue to have the table, and it should continue to filter out the
> other columns that were being filtered out, regardless of CASCADE/RESTRICT.
> However, if the column is *included* in the publication, and you drop
> it, ISTM there are two cases:
> 
> 1. If it's DROP CASCADE, then the list of columns to replicate should
> continue to have all columns it previously had, so just remove the
> column that is being dropped.
> 
> 2. If it's DROP RESTRICT, then an error should be raised so that the
> user can make a concious decision to remove the column from the filter
> before dropping the column.
> 

FWIW I think this is a sensible behavior.

I don't quite see why dropping a column should remove the table from
publication (assuming there are some columns still replicated).

Of course, it may break the subscriber (e.g. when there was NOT NULL
constraint on that column), but DROP RESTRICT (which I assume is the
default mode) prevents that. And if DROP CASCADE is specified, I think
it's reasonable to require the user to fix the fallout.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  33. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2021-09-07T00:29:49Z

    Here are some v5 review comments for your consideration:

------

1. src/backend/access/common/relation.c

@@ -215,3 +217,22 @@ relation_close(Relation relation, LOCKMODE lockmode)
  if (lockmode != NoLock)
  UnlockRelationId(&relid, lockmode);
 }
+
+/*
+ * Return a bitmapset of attributes given the list of column names
+ */
+Bitmapset*
+get_table_columnset(Oid relid, List *columns, Bitmapset *att_map)
+{


IIUC that 3rd parameter (att_map) is always passed as NULL to
get_table_columnset function because you are constructing this
Bitmapset from scratch. Maybe I am mistaken, but if not then what is
the purpose of that att_map parameter?

------

2. src/backend/catalog/pg_publication.c

+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Column filter must include REPLICA IDENTITY columns")));

Is ERRCODE_INVALID_COLUMN_REFERENCE a more appropriate errcode to use here?

------

3. src/backend/catalog/pg_publication.c

+ else
+ {
+ Bitmapset *filtermap = NULL;
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
INDEX_ATTR_BITMAP_IDENTITY_KEY);

The RelationGetIndexAttrBitmap function comment says "should be
bms_free'd when not needed anymore" but it seems the patch code is not
freeing idattrs when finished using it.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  34. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-07T05:00:18Z

    On Mon, Sep 6, 2021 at 9:25 PM Rahila Syed <rahilasyed90@gmail.com> wrote:
>
> On Mon, Sep 6, 2021 at 8:53 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>
>> Did you give any thoughts to my earlier suggestion related to syntax [1]?
>>
>>
>> [1] - https://www.postgresql.org/message-id/CAA4eK1J9b_0_PMnJ2jq9E55bcbmTKdUmy6jPnkf1Zwy2jxah_g%40mail.gmail.com
>
>
> For future support to replicate all columns except (x,y,z), I think some optional keywords like
> COLUMNS NOT IN can be inserted between table name and (*columns_list*) as follows.
> ALTER PUBLICATION ADD TABLE tab_name [COLUMNS NOT IN] (x,y,z)
> I think this should be possible as a future addition to proposed syntax in the patch.
> Please let me know your opinion.
>

Right, I don't want you to implement that feature as part of this
patch but how about using COLUMNS or similar keyword in column filter
like ALTER PUBLICATION ADD TABLE tab_name COLUMNS (c1, c2, ...)? This
can make it easier to extend in the future.


-- 
With Regards,
Amit Kapila.

  35. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-07T05:35:54Z

    On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-06, Rahila Syed wrote:
>
> > > > ... ugh.  Since CASCADE is already defined to be a
> > > > potentially-data-loss operation, then that may be acceptable
> > > > behavior.  For sure the default RESTRICT behavior shouldn't do it,
> > > > though.
> > >
> > > That makes sense to me.
> >
> > However, the default (RESTRICT) behaviour of DROP TABLE allows
> > removing the table from the publication. I have implemented the
> > removal of table from publication on drop column (RESTRICT)  on the
> > same lines.
>
> But dropping the table is quite a different action from dropping a
> column, isn't it?  If you drop a table, it seems perfectly reasonable
> that it has to be removed from the publication -- essentially, when the
> user drops a table, she is saying "I don't care about this table
> anymore".  However, if you drop just one column, that doesn't
> necessarily mean that the user wants to stop publishing the whole table.
> Removing the table from the publication in ALTER TABLE DROP COLUMN seems
> like an overreaction.  (Except perhaps in the special case were the
> column being dropped is the only one that was being published.)
>
> So let's discuss what should happen.  If you drop a column, and the
> column is filtered out, then it seems to me that the publication should
> continue to have the table, and it should continue to filter out the
> other columns that were being filtered out, regardless of CASCADE/RESTRICT.
>

Yeah, for this case we don't need to do anything and I am not sure if
the patch is dropping tables in this case?

> However, if the column is *included* in the publication, and you drop
> it, ISTM there are two cases:
>
> 1. If it's DROP CASCADE, then the list of columns to replicate should
> continue to have all columns it previously had, so just remove the
> column that is being dropped.
>

Note that for a somewhat similar case in the index (where the index
has an expression) we drop the index if one of the columns used in the
index expression is dropped, so we might want to just remove the
entire filter here instead of just removing the particular column or
remove the entire table from publication as Rahila is proposing.

I think removing just a particular column can break the replication
for Updates and Deletes if the removed column is part of replica
identity. If the entire filter is removed then also the entire
replication can break, so, I think Rahila's proposal is worth
considering.

> 2. If it's DROP RESTRICT, then an error should be raised so that the
> user can make a concious decision to remove the column from the filter
> before dropping the column.
>

I think one can argue for a similar case for index. If we are allowing
the index to be dropped even with RESTRICT then why not column filter?

> > Did you give any thoughts to my earlier suggestion related to syntax [1]?
> >
> > [1] https://www.postgresql.org/message-id/CAA4eK1J9b_0_PMnJ2jq9E55bcbmTKdUmy6jPnkf1Zwy2jxah_g%40mail.gmail.com
>
> This is a great followup idea, after the current feature is committed.
>

As mentioned in my response to Rahila, I was just thinking of using an
optional keyword Column for column filter so that we can extend it
later.

-- 
With Regards,
Amit Kapila.

  36. Re: Column Filtering in Logical Replication

    Dilip Kumar <dilipbalaut@gmail.com> — 2021-09-07T05:56:17Z

    On Tue, Sep 7, 2021 at 11:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

> On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
> wrote:
> >
> > On 2021-Sep-06, Rahila Syed wrote:
> >
> > > > > ... ugh.  Since CASCADE is already defined to be a
> > > > > potentially-data-loss operation, then that may be acceptable
> > > > > behavior.  For sure the default RESTRICT behavior shouldn't do it,
> > > > > though.
> > > >
> > > > That makes sense to me.
> > >
> > > However, the default (RESTRICT) behaviour of DROP TABLE allows
> > > removing the table from the publication. I have implemented the
> > > removal of table from publication on drop column (RESTRICT)  on the
> > > same lines.
> >
> > But dropping the table is quite a different action from dropping a
> > column, isn't it?  If you drop a table, it seems perfectly reasonable
> > that it has to be removed from the publication -- essentially, when the
> > user drops a table, she is saying "I don't care about this table
> > anymore".  However, if you drop just one column, that doesn't
> > necessarily mean that the user wants to stop publishing the whole table.
> > Removing the table from the publication in ALTER TABLE DROP COLUMN seems
> > like an overreaction.  (Except perhaps in the special case were the
> > column being dropped is the only one that was being published.)
> >
> > So let's discuss what should happen.  If you drop a column, and the
> > column is filtered out, then it seems to me that the publication should
> > continue to have the table, and it should continue to filter out the
> > other columns that were being filtered out, regardless of
> CASCADE/RESTRICT.
> >
>
> Yeah, for this case we don't need to do anything and I am not sure if
> the patch is dropping tables in this case?
>
> > However, if the column is *included* in the publication, and you drop
> > it, ISTM there are two cases:
> >
> > 1. If it's DROP CASCADE, then the list of columns to replicate should
> > continue to have all columns it previously had, so just remove the
> > column that is being dropped.
> >
>
> Note that for a somewhat similar case in the index (where the index
> has an expression) we drop the index if one of the columns used in the
> index expression is dropped, so we might want to just remove the
> entire filter here instead of just removing the particular column or
> remove the entire table from publication as Rahila is proposing.
>
> I think removing just a particular column can break the replication
> for Updates and Deletes if the removed column is part of replica
> identity.
>

But how this is specific to this patch, I think the behavior should be the
same as what is there now, I mean now also we can drop the columns which
are part of replica identity right.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  37. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-07T06:43:08Z

    On Tue, Sep 7, 2021 at 11:26 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Sep 7, 2021 at 11:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>> >
>> > On 2021-Sep-06, Rahila Syed wrote:
>> >
>> > > > > ... ugh.  Since CASCADE is already defined to be a
>> > > > > potentially-data-loss operation, then that may be acceptable
>> > > > > behavior.  For sure the default RESTRICT behavior shouldn't do it,
>> > > > > though.
>> > > >
>> > > > That makes sense to me.
>> > >
>> > > However, the default (RESTRICT) behaviour of DROP TABLE allows
>> > > removing the table from the publication. I have implemented the
>> > > removal of table from publication on drop column (RESTRICT)  on the
>> > > same lines.
>> >
>> > But dropping the table is quite a different action from dropping a
>> > column, isn't it?  If you drop a table, it seems perfectly reasonable
>> > that it has to be removed from the publication -- essentially, when the
>> > user drops a table, she is saying "I don't care about this table
>> > anymore".  However, if you drop just one column, that doesn't
>> > necessarily mean that the user wants to stop publishing the whole table.
>> > Removing the table from the publication in ALTER TABLE DROP COLUMN seems
>> > like an overreaction.  (Except perhaps in the special case were the
>> > column being dropped is the only one that was being published.)
>> >
>> > So let's discuss what should happen.  If you drop a column, and the
>> > column is filtered out, then it seems to me that the publication should
>> > continue to have the table, and it should continue to filter out the
>> > other columns that were being filtered out, regardless of CASCADE/RESTRICT.
>> >
>>
>> Yeah, for this case we don't need to do anything and I am not sure if
>> the patch is dropping tables in this case?
>>
>> > However, if the column is *included* in the publication, and you drop
>> > it, ISTM there are two cases:
>> >
>> > 1. If it's DROP CASCADE, then the list of columns to replicate should
>> > continue to have all columns it previously had, so just remove the
>> > column that is being dropped.
>> >
>>
>> Note that for a somewhat similar case in the index (where the index
>> has an expression) we drop the index if one of the columns used in the
>> index expression is dropped, so we might want to just remove the
>> entire filter here instead of just removing the particular column or
>> remove the entire table from publication as Rahila is proposing.
>>
>> I think removing just a particular column can break the replication
>> for Updates and Deletes if the removed column is part of replica
>> identity.
>
>
> But how this is specific to this patch, I think the behavior should be the same as what is there now, I mean now also we can drop the columns which are part of replica identity right.
>

Sure, but we drop replica identity and corresponding index as well.
The patch ensures that replica identity columns must be part of the
column filter and now that restriction won't hold anymore. I think if
we want to retain that restriction then it is better to either remove
the entire filter or remove the entire table. Anyway, the main point
was that if we can remove the index/replica identity, it seems like
there should be the same treatment for column filter.

Another related point that occurred to me is that if the user changes
replica identity then probably we should ensure that the column
filters for the table still holds the creteria or maybe we need to
remove the filter in that case as well. I am not sure if the patch is
already doing something about it and if not then isn't it better to do
something about it?

-- 
With Regards,
Amit Kapila.

  38. Re: Column Filtering in Logical Replication

    Euler Taveira <euler@eulerto.com> — 2021-09-07T22:39:30Z

    On Mon, Sep 6, 2021, at 2:51 PM, Alvaro Herrera wrote:
> I pushed the clerical part of this -- namely the addition of
> PublicationTable node and PublicationRelInfo struct.  I attach the part
> of your v4 patch that I didn't include.  It contains a couple of small
> corrections, but I didn't do anything invasive (such as pgindent)
> because that would perhaps cause you too much merge pain.
While updating the row filter patch [1] (because it also uses these
structures), I noticed that you use PublicationRelInfo as a type name instead
of PublicationRelationInfo. I choose the latter because there is already a data
structure named PublicationRelInfo (pg_dump.h). It is a client-side data
structure but I doesn't seem a good practice to duplicate data structure names
over the same code base.

[1] https://www.postgresql.org/message-id/0c2464d4-65f4-4d91-aeb2-c5584c1350f5%40www.fastmail.com


--
Euler Taveira
EDB   https://www.enterprisedb.com/

  39. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-15T03:34:21Z

    On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> I pushed the clerical part of this -- namely the addition of
> PublicationTable node and PublicationRelInfo struct.
>

One point to note here is that we are developing a generic grammar for
publications where not only tables but other objects like schema,
sequences, etc. can be specified, see [1]. So, there is some overlap
in the grammar modifications being made by this patch and the work
being done in that other thread. As both the patches are being
developed at the same time, it might be better to be in sync,
otherwise, some of the work needs to be changed. I can see that in the
patch [2] (v28-0002-Added-schema-level-support-for-publication) being
developed there the changes made by the above commit needs to be
changed again to represent a generic object for publication. It is
possible that we can do it some other way but I think it would be
better to coordinate the work in both threads. The other approach is
to continue independently and the later patch can adapt to the earlier
one which is fine too but it might be more work for the later one.

[1] - https://www.postgresql.org/message-id/877603.1629120678%40sss.pgh.pa.us
[2] - postgresql.org/message-id/CALDaNm0OudeDeFN7bSWPro0hgKx%3D1zPgcNFWnvU_G6w3mDPX0Q%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  40. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-15T11:50:07Z

    On 2021-Sep-15, Amit Kapila wrote:

> On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> >
> > I pushed the clerical part of this -- namely the addition of
> > PublicationTable node and PublicationRelInfo struct.
> 
> One point to note here is that we are developing a generic grammar for
> publications where not only tables but other objects like schema,
> sequences, etc. can be specified, see [1]. So, there is some overlap
> in the grammar modifications being made by this patch and the work
> being done in that other thread.

Oh rats.  I was not aware of that thread, or indeed of the fact that
adding multiple object types to publications was being considered.

I do see that 0002 there contains gram.y changes, but AFAICS those
changes don't allow specifying a column list for a table, so there are
some changes needed in that patch for that either way.

I agree that it's better to move forward in unison.

I noticed that 0002 in that other patch uses a void * pointer in
PublicationObjSpec that "could be either RangeVar or String", which
strikes me as a really bad idea.  (Already discussed in some other
thread recently, maybe this one or the row filtering one.)

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/

  41. Re: Column Filtering in Logical Replication

    vignesh C <vignesh21@gmail.com> — 2021-09-15T12:19:19Z

    On Wed, Sep 15, 2021 at 5:20 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-15, Amit Kapila wrote:
>
> > On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> > >
> > > I pushed the clerical part of this -- namely the addition of
> > > PublicationTable node and PublicationRelInfo struct.
> >
> > One point to note here is that we are developing a generic grammar for
> > publications where not only tables but other objects like schema,
> > sequences, etc. can be specified, see [1]. So, there is some overlap
> > in the grammar modifications being made by this patch and the work
> > being done in that other thread.
>
> Oh rats.  I was not aware of that thread, or indeed of the fact that
> adding multiple object types to publications was being considered.
>
> I do see that 0002 there contains gram.y changes, but AFAICS those
> changes don't allow specifying a column list for a table, so there are
> some changes needed in that patch for that either way.
>
> I agree that it's better to move forward in unison.
>
> I noticed that 0002 in that other patch uses a void * pointer in
> PublicationObjSpec that "could be either RangeVar or String", which
> strikes me as a really bad idea.  (Already discussed in some other
> thread recently, maybe this one or the row filtering one.)

I have extracted the parser code and attached it here, so that it will
be easy to go through. We wanted to support the following syntax as in
[1]:
CREATE PUBLICATION pub1 FOR
TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2,
SEQUENCE seq1,seq2, ALL SEQUENCES IN SCHEMA s3,s4;

Columns can be added to PublicationObjSpec data structure. The patch
Generic_object_type_parser_002_table_schema_publication.patch has the
changes that were used to handle the parsing. Schema and Relation both
are different objects, schema is of string type and relation is of
RangeVar type. While parsing, schema name is parsed in string format
and relation is parsed and converted to rangevar type, these objects
will be then handled accordingly during post processing. That is the
reason it used void * type which could hold both RangeVar and String.
Thoughts?

[1] - https://www.postgresql.org/message-id/877603.1629120678%40sss.pgh.pa.us

Regards,
Vignesh

  42. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-15T12:36:51Z

    On 2021-Sep-15, vignesh C wrote:

> I have extracted the parser code and attached it here, so that it will
> be easy to go through. We wanted to support the following syntax as in
> [1]:
> CREATE PUBLICATION pub1 FOR
> TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2,
> SEQUENCE seq1,seq2, ALL SEQUENCES IN SCHEMA s3,s4;

Oh, thanks, it looks like this can be useful.  We can get the common
grammar done and then rebase all the other patches (I was also just told
about support for sequences in [1]) on top.

[1] https://postgr.es/m/3d6df331-5532-6848-eb45-344b265e0238@enterprisedb.com

> Columns can be added to PublicationObjSpec data structure.

Right.  (As a List of String, I imagine.)

> The patch
> Generic_object_type_parser_002_table_schema_publication.patch has the
> changes that were used to handle the parsing. Schema and Relation both
> are different objects, schema is of string type and relation is of
> RangeVar type. While parsing, schema name is parsed in string format
> and relation is parsed and converted to rangevar type, these objects
> will be then handled accordingly during post processing.

Yeah, I think it'd be cleaner if the node type has two members, something like
this

typedef struct PublicationObjSpec
{
	NodeTag		type;
	PublicationObjSpecType pubobjtype;	/* type of this publication object */
	RangeVar	*rv;		/* if a table */
	String		*objname;	/* if a schema */
	int		location;		/* token location, or -1 if unknown */
} PublicationObjSpec;

and only one of them is set, the other is NULL, depending on the object type.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/

  43. Re: Column Filtering in Logical Replication

    Euler Taveira <euler@eulerto.com> — 2021-09-15T14:46:34Z

    On Wed, Sep 15, 2021, at 9:19 AM, vignesh C wrote:
> I have extracted the parser code and attached it here, so that it will
> be easy to go through. We wanted to support the following syntax as in
> [1]:
> CREATE PUBLICATION pub1 FOR
> TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2,
> SEQUENCE seq1,seq2, ALL SEQUENCES IN SCHEMA s3,s4;
I don't like this syntax. It seems too much syntax for the same purpose in a
single command. If you look at GRANT command whose ALL TABLES IN SCHEMA syntax
was extracted, you can use ON TABLE or ON ALL TABLES IN SCHEMA; you cannot use
both. This proposal allows duplicate objects (of course, you can ignore it but
the current code prevent duplicates -- see publication_add_relation).

IMO you should mimic the GRANT grammar and have multiple commands for row
filtering, column filtering, and ALL FOO IN SCHEMA. The filtering patches only
use the FOR TABLE syntax. The later won't have filtering syntax. Having said
that the grammar should be:

CREATE PUBLICATION name
    [ FOR TABLE [ ONLY ] table_name [ * ] [ (column_name [, ...] ) ] [ WHERE (expression) ] [, ...]
      | FOR ALL TABLES
      | FOR ALL TABLES IN SCHEMA schema_name, [, ...]
      | FOR ALL SEQUENCES IN SCHEMA schema_name, [, ...] ]
    [ WITH ( publication_parameter [= value] [, ... ] ) ]

ALTER PUBLICATION name ADD TABLE [ ONLY ] table_name [ * ] [ (column_name [, ...] ) ] [ WHERE (expression) ]
ALTER PUBLICATION name ADD ALL TABLES IN SCHEMA schema_name, [, ...]
ALTER PUBLICATION name ADD ALL SEQUENCES IN SCHEMA schema_name, [, ...]

ALTER PUBLICATION name SET TABLE [ ONLY ] table_name [ * ] [ (column_name [, ...] ) ] [ WHERE (expression) ]
ALTER PUBLICATION name SET ALL TABLES IN SCHEMA schema_name, [, ...]
ALTER PUBLICATION name SET ALL SEQUENCES IN SCHEMA schema_name, [, ...]

ALTER PUBLICATION name DROP TABLE [ ONLY ] table_name [ * ]
ALTER PUBLICATION name DROP ALL TABLES IN SCHEMA schema_name, [, ...]
ALTER PUBLICATION name DROP ALL SEQUENCES IN SCHEMA schema_name, [, ...]

Opinions?


--
Euler Taveira
EDB   https://www.enterprisedb.com/

  44. RE: Column Filtering in Logical Replication

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2021-09-16T02:06:09Z

    On Wednesday, September 15, 2021 8:19 PM vignesh C <vignesh21@gmail.com> wrote:
> I have extracted the parser code and attached it here, so that it will be easy to
> go through. We wanted to support the following syntax as in
> [1]:
> CREATE PUBLICATION pub1 FOR
> TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2, SEQUENCE seq1,seq2, ALL
> SEQUENCES IN SCHEMA s3,s4;

I am +1 for this syntax.

This syntax is more flexible than adding or dropping different type objects in
separate commands. User can either use one single command to add serval different
objects or use serval commands to add each type objects.

Best regards,
Hou zj

  45. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2021-09-16T03:06:58Z

    On Tue, Sep 7, 2021 at 3:51 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
...
> I pushed the clerical part of this -- namely the addition of
> PublicationTable node and PublicationRelInfo struct.  I attach the part
> of your v4 patch that I didn't include.  It contains a couple of small
> corrections, but I didn't do anything invasive (such as pgindent)
> because that would perhaps cause you too much merge pain.

I noticed that the latest v5 no longer includes the TAP test which was
in the v4 patch.

(src/test/subscription/t/021_column_filter.pl)

Was that omission deliberate?

------
Kind Regards,
Peter Smith.
Fujitsu Australia.

  46. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-16T03:15:15Z

    On Wed, Sep 15, 2021 at 6:06 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-15, vignesh C wrote:
> > The patch
> > Generic_object_type_parser_002_table_schema_publication.patch has the
> > changes that were used to handle the parsing. Schema and Relation both
> > are different objects, schema is of string type and relation is of
> > RangeVar type. While parsing, schema name is parsed in string format
> > and relation is parsed and converted to rangevar type, these objects
> > will be then handled accordingly during post processing.
>
> Yeah, I think it'd be cleaner if the node type has two members, something like
> this
>
> typedef struct PublicationObjSpec
> {
>         NodeTag         type;
>         PublicationObjSpecType pubobjtype;      /* type of this publication object */
>         RangeVar        *rv;            /* if a table */
>         String          *objname;       /* if a schema */
>         int             location;               /* token location, or -1 if unknown */
> } PublicationObjSpec;
>
> and only one of them is set, the other is NULL, depending on the object type.
>

I think the problem here is that with the proposed grammar we won't be
always able to distinguish names at the gram.y stage. Some post
parsing analysis is required to attribute the right type to name as is
done in the patch. The same seems to be indicated by Tom in his email
as well where he has proposed this syntax [1]. Also, something similar
is done for privilege_target (GRANT syntax) where we have a list of
objects but here the story is slightly more advanced because we are
planning to allow specifying multiple objects in one command. One
might think that we can identify each type of objects lists separately
but that gives grammar conflicts as it is not able to identify whether
the comma ',' is used for the same type object or for the next type.
Due to which we need to come up with a generic object for names to
which we attribute the right type in post parse analysis. Now, I think
instead of void *, it might be better to use Node * for generic
objects unless we have some problem.

[1] - https://www.postgresql.org/message-id/877603.1629120678%40sss.pgh.pa.us

-- 
With Regards,
Amit Kapila.

  47. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-16T03:22:56Z

    On Wed, Sep 15, 2021 at 8:19 PM Euler Taveira <euler@eulerto.com> wrote:
>
> On Wed, Sep 15, 2021, at 9:19 AM, vignesh C wrote:
>
> I have extracted the parser code and attached it here, so that it will
> be easy to go through. We wanted to support the following syntax as in
> [1]:
> CREATE PUBLICATION pub1 FOR
> TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2,
> SEQUENCE seq1,seq2, ALL SEQUENCES IN SCHEMA s3,s4;
>
> I don't like this syntax. It seems too much syntax for the same purpose in a
> single command. If you look at GRANT command whose ALL TABLES IN SCHEMA syntax
> was extracted, you can use ON TABLE or ON ALL TABLES IN SCHEMA; you cannot use
> both. This proposal allows duplicate objects (of course, you can ignore it but
> the current code prevent duplicates -- see publication_add_relation).
>
> IMO you should mimic the GRANT grammar and have multiple commands for row
> filtering, column filtering, and ALL FOO IN SCHEMA. The filtering patches only
> use the FOR TABLE syntax. The later won't have filtering syntax.
>

Sure, but we don't prevent if the user uses only FOR TABLE variant.
OTOH, it is better to provide flexibility to allow multiple objects in
one command unless that is not feasible. It saves the effort of users
in many cases. In short, +1 for the syntax where multiple objects can
be allowed.

-- 
With Regards,
Amit Kapila.

  48. Re: Column Filtering in Logical Replication

    vignesh C <vignesh21@gmail.com> — 2021-09-16T05:05:21Z

    On Thu, Sep 16, 2021 at 8:45 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Sep 15, 2021 at 6:06 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> >
> > On 2021-Sep-15, vignesh C wrote:
> > > The patch
> > > Generic_object_type_parser_002_table_schema_publication.patch has the
> > > changes that were used to handle the parsing. Schema and Relation both
> > > are different objects, schema is of string type and relation is of
> > > RangeVar type. While parsing, schema name is parsed in string format
> > > and relation is parsed and converted to rangevar type, these objects
> > > will be then handled accordingly during post processing.
> >
> > Yeah, I think it'd be cleaner if the node type has two members, something like
> > this
> >
> > typedef struct PublicationObjSpec
> > {
> >         NodeTag         type;
> >         PublicationObjSpecType pubobjtype;      /* type of this publication object */
> >         RangeVar        *rv;            /* if a table */
> >         String          *objname;       /* if a schema */
> >         int             location;               /* token location, or -1 if unknown */
> > } PublicationObjSpec;
> >
> > and only one of them is set, the other is NULL, depending on the object type.
> >
>
> I think the problem here is that with the proposed grammar we won't be
> always able to distinguish names at the gram.y stage.

This is the issue that Amit was talking about:
gram.y: error: shift/reduce conflicts: 2 found, 0 expected
gram.y: warning: shift/reduce conflict on token ',' [-Wcounterexamples]
  First example: CREATE PUBLICATION name FOR TABLE relation_expr_list
• ',' relation_expr ',' PublicationObjSpec opt_definition $end
  Shift derivation
    $accept
    ↳ parse_toplevel
                                                              $end
      ↳ stmtmulti
        ↳ toplevel_stmt
          ↳ stmt
            ↳ CreatePublicationStmt
              ↳ CREATE PUBLICATION name FOR pub_obj_list
                                               opt_definition
                                            ↳ PublicationObjSpec
                        ',' PublicationObjSpec
                                              ↳ TABLE relation_expr_list
                                                      ↳
relation_expr_list • ',' relation_expr
  Second example: CREATE PUBLICATION name FOR TABLE relation_expr_list
• ',' PublicationObjSpec opt_definition $end
  Reduce derivation
    $accept
    ↳ parse_toplevel
                                            $end
      ↳ stmtmulti
        ↳ toplevel_stmt
          ↳ stmt
            ↳ CreatePublicationStmt
              ↳ CREATE PUBLICATION name FOR pub_obj_list
                             opt_definition
                                            ↳ pub_obj_list
      ',' PublicationObjSpec
                                              ↳ PublicationObjSpec
                                                ↳ TABLE relation_expr_list •
Here it is not able to distinguish if ',' is used for the next table
name or the next object.
I was able to reproduce this issue with the attached patch.

Regards,
Vignesh

  49. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-16T12:44:35Z

    On 2021-Sep-16, Amit Kapila wrote:

> I think the problem here is that with the proposed grammar we won't be
> always able to distinguish names at the gram.y stage. Some post
> parsing analysis is required to attribute the right type to name as is
> done in the patch.

Doesn't it work to stuff them all into RangeVars?  Then you don't need
to make the node type a monstrosity, just bail out in parse analysis if
an object spec has more elements in the RV than the object type allows.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/

  50. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-16T13:48:29Z

    On Thu, Sep 16, 2021 at 6:14 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-16, Amit Kapila wrote:
>
> > I think the problem here is that with the proposed grammar we won't be
> > always able to distinguish names at the gram.y stage. Some post
> > parsing analysis is required to attribute the right type to name as is
> > done in the patch.
>
> Doesn't it work to stuff them all into RangeVars?  Then you don't need
> to make the node type a monstrosity, just bail out in parse analysis if
> an object spec has more elements in the RV than the object type allows.
>

So, are you suggesting that we store even schema names corresponding
to FOR ALL TABLES IN SCHEMA s1 [, ...] grammar in RangeVars in some
way (say store schema name in relname or schemaname field of RangeVar)
at gram.y stage and then later extract it from RangeVar? If so, why do
you think it would be better than the current proposed way?

-- 
With Regards,
Amit Kapila.

  51. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-16T13:50:25Z

    On 2021-Sep-16, vignesh C wrote:

> diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
> index e3068a374e..c50bb570ea 100644
> --- a/src/backend/parser/gram.y
> +++ b/src/backend/parser/gram.y

Yeah, on a quick glance this looks all wrong.  Your PublicationObjSpec
production should return a node with tag PublicationObjSpec, and
pubobj_expr should not exist at all -- that stuff is just making it all
more confusing.

I think it'd be something like this:

PublicationObjSpec:	
			ALL TABLES
					{
						$$ = makeNode(PublicationObjSpec);
						$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES;
						$$->location = @1;
					}
			| TABLE qualified_name
					{
						$$ = makeNode(PublicationObjSpec);
						$$->pubobjtype = PUBLICATIONOBJ_TABLE;
						$$->pubobj = $2;
						$$->location = @1;
					}
			| ALL TABLES IN_P SCHEMA name
					{
						$$ = makeNode(PublicationObjSpec);
						$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES_IN_SCHEMA;
						$$->pubobj = makeRangeVar( ... $5 ... );
						$$->location = @1;
					}
			| qualified_name
					{
						$$ = makeNode(PublicationObjSpec);
						$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
						$$->pubobj = $1;
						$$->location = @1;
					};

You need a single object name under TABLE, not a list -- this was Tom's
point about needing post-processing to determine how to assign a type to
a object that's what I named PUBLICATIONOBJ_CONTINUATION here.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Puedes vivir sólo una vez, pero si lo haces bien, una vez es suficiente"

  52. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-16T14:36:32Z

    On 2021-Sep-16, Alvaro Herrera wrote:

Actually, something like this might be better:

> PublicationObjSpec:	

> 			| TABLE qualified_name
> 					{
> 						$$ = makeNode(PublicationObjSpec);
> 						$$->pubobjtype = PUBLICATIONOBJ_TABLE;
> 						$$->pubrvobj = $2;
> 						$$->location = @1;
> 					}
> 			| ALL TABLES IN_P SCHEMA name
> 					{
> 						$$ = makeNode(PublicationObjSpec);
> 						$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES_IN_SCHEMA;
> 						$$->pubplainobj = $5;
> 						$$->location = @1;
> 					}

So you don't have to cram the schema name in a RangeVar, which would
indeed be quite awkward.  (I'm sure you can come up with better names
for the struct members there ...)

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Porque francamente, si para saber manejarse a uno mismo hubiera que
rendir examen... ¿Quién es el machito que tendría carnet?"  (Mafalda)

  53. Re: Column Filtering in Logical Replication

    vignesh C <vignesh21@gmail.com> — 2021-09-17T04:06:41Z

    On Thu, Sep 16, 2021 at 7:20 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-16, vignesh C wrote:
>
> > diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
> > index e3068a374e..c50bb570ea 100644
> > --- a/src/backend/parser/gram.y
> > +++ b/src/backend/parser/gram.y
>
> Yeah, on a quick glance this looks all wrong.  Your PublicationObjSpec
> production should return a node with tag PublicationObjSpec, and
> pubobj_expr should not exist at all -- that stuff is just making it all
> more confusing.
>
> I think it'd be something like this:
>
> PublicationObjSpec:
>                         ALL TABLES
>                                         {
>                                                 $$ = makeNode(PublicationObjSpec);
>                                                 $$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES;
>                                                 $$->location = @1;
>                                         }
>                         | TABLE qualified_name
>                                         {
>                                                 $$ = makeNode(PublicationObjSpec);
>                                                 $$->pubobjtype = PUBLICATIONOBJ_TABLE;
>                                                 $$->pubobj = $2;
>                                                 $$->location = @1;
>                                         }
>                         | ALL TABLES IN_P SCHEMA name
>                                         {
>                                                 $$ = makeNode(PublicationObjSpec);
>                                                 $$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES_IN_SCHEMA;
>                                                 $$->pubobj = makeRangeVar( ... $5 ... );
>                                                 $$->location = @1;
>                                         }
>                         | qualified_name
>                                         {
>                                                 $$ = makeNode(PublicationObjSpec);
>                                                 $$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
>                                                 $$->pubobj = $1;
>                                                 $$->location = @1;
>                                         };
>
> You need a single object name under TABLE, not a list -- this was Tom's
> point about needing post-processing to determine how to assign a type to
> a object that's what I named PUBLICATIONOBJ_CONTINUATION here.

In the above, we will not be able to use qualified_name, as
qualified_name will not support the following syntaxes:
create publication pub1 for table t1 *;
create publication pub1 for table ONLY t1 *;
create publication pub1 for table ONLY (t1);

To solve this problem we can change qualified_name to relation_expr
but the problem with doing that is that the user will be able to
provide the following syntaxes:
create publication pub1 for all tables in schema sch1 *;
create publication pub1 for all tables in schema ONLY sch1 *;
create publication pub1 for all tables in schema ONLY (sch1);

To handle this we will need some special flag which will differentiate
these and throw errors at post processing time. We need to define an
expression similar to relation_expr say pub_expr which handles all
variants of qualified_name and then use a special flag so that we can
throw an error if somebody uses the above type of syntax for schema
names. And then if we have to distinguish between schema name and
relation name variant, then we need few other things.

We proposed the below solution which handles all these problems and
also used Node type which need not store schemaname in RangeVar type:
pubobj_expr:
                        pubobj_name
                                {
                                        /* inheritance query, implicitly */
                                        $$ = makeNode(PublicationObjSpec);
                                        $$->object = $1;
                                }
                        | extended_relation_expr
                                {
                                        $$ = makeNode(PublicationObjSpec);
                                        $$->object = (Node *)$1;
                                }
                        | CURRENT_SCHEMA
                                {
                                        $$ = makeNode(PublicationObjSpec);
                                        $$->object = (Node
*)makeString("CURRENT_SCHEMA");
                                }
                ;
/* This can be either a schema or relation name. */
pubobj_name:
                        ColId
                                {
                                        $$ = (Node *) makeString($1);
                                }
                        | ColId indirection
                                {
                                        $$ = (Node *)
makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
                                }
                ;
/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
PublicationObjSpec:     TABLE pubobj_expr
                                        {
                                                $$ = $2;
                                                $$->pubobjtype =
PUBLICATIONOBJ_TABLE;
                                                $$->location = @1;
                                        }
                        | ALL TABLES IN_P SCHEMA pubobj_expr
                                        {
                                                $$ = $5;
                                                $$->pubobjtype =
PUBLICATIONOBJ_REL_IN_SCHEMA;
                                                $$->location = @1;
                                        }
                        | pubobj_expr
                                        {
                                                $$ = $1;
                                                $$->pubobjtype =
PUBLICATIONOBJ_UNKNOWN;
                                                $$->location = @1;
                                        }
                ;

The same has been proposed in the recent version of patch [1].
[1] - https://www.postgresql.org/message-id/CALDaNm0OudeDeFN7bSWPro0hgKx%3D1zPgcNFWnvU_G6w3mDPX0Q%40mail.gmail.com
Thoughts?

Regards,
Vignesh

  54. RE: Column Filtering in Logical Replication

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2021-09-17T05:08:36Z

    On Thurs, Sep 16, 2021 10:37 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> On 2021-Sep-16, Alvaro Herrera wrote:
> 
> Actually, something like this might be better:
> 
> > PublicationObjSpec:
> 
> >			| TABLE qualified_name
> >					{
> >						$$ = makeNode(PublicationObjSpec);
> >						$$->pubobjtype = PUBLICATIONOBJ_TABLE;
> >						$$->pubrvobj = $2;
> >						$$->location = @1;
> >					}
> >			| ALL TABLES IN_P SCHEMA name
> >					{
> >						$$ = makeNode(PublicationObjSpec);
> >						$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES_IN_SCHEMA;
> >						$$->pubplainobj = $5;
> >						$$->location = @1;
> >					}
> So you don't have to cram the schema name in a RangeVar, which would indeed
> be quite awkward.  (I'm sure you can come up with better names for the struct
> members there ...)> 

Did you mean something like the following ?
-----
PublicationObjSpec:
		TABLE qualified_name {...}
		| ALL TABLES IN_P SCHEMA name {...}
		;

pub_obj_list:
		PublicationObjSpec
        | pub_obj_list ',' PublicationObjSpec
-----

If so, I think it only supports syntaxes like "TABLE a, TABLE b, TABLE c" while
we cannnot use "TABLE a,b,c". To support multiple objects, we need a bare name
in PublicationObjSpec.

Or Did you mean something like this ?
-----
PublicationObjSpec:
		TABLE qualified_name {...}
		| ALL TABLES IN_P SCHEMA name {...}
		| qualified_name {...}
		;
-----

I think this doesn't support relation expression like "table */ONLY table/ONLY
(table)" as memtioned by Vignesh [1].

Thoughts ?

[1] https://www.postgresql.org/message-id/CALDaNm06%3DLDytYyY%2BxcAQd8UK_YpJ3zMo4P5V8KBArw6MoDWDg%40mail.gmail.com

Best regards,
Hou zj

  55. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-23T10:18:57Z

    On Fri, Sep 17, 2021 at 9:36 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, Sep 16, 2021 at 7:20 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> >
> > On 2021-Sep-16, vignesh C wrote:
> >
> > > diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
> > > index e3068a374e..c50bb570ea 100644
> > > --- a/src/backend/parser/gram.y
> > > +++ b/src/backend/parser/gram.y
> >
> > Yeah, on a quick glance this looks all wrong.  Your PublicationObjSpec
> > production should return a node with tag PublicationObjSpec, and
> > pubobj_expr should not exist at all -- that stuff is just making it all
> > more confusing.
> >
> > I think it'd be something like this:
> >
> > PublicationObjSpec:
> >                         ALL TABLES
> >                                         {
> >                                                 $$ = makeNode(PublicationObjSpec);
> >                                                 $$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES;
> >                                                 $$->location = @1;
> >                                         }
> >                         | TABLE qualified_name
> >                                         {
> >                                                 $$ = makeNode(PublicationObjSpec);
> >                                                 $$->pubobjtype = PUBLICATIONOBJ_TABLE;
> >                                                 $$->pubobj = $2;
> >                                                 $$->location = @1;
> >                                         }
> >                         | ALL TABLES IN_P SCHEMA name
> >                                         {
> >                                                 $$ = makeNode(PublicationObjSpec);
> >                                                 $$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES_IN_SCHEMA;
> >                                                 $$->pubobj = makeRangeVar( ... $5 ... );
> >                                                 $$->location = @1;
> >                                         }
> >                         | qualified_name
> >                                         {
> >                                                 $$ = makeNode(PublicationObjSpec);
> >                                                 $$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
> >                                                 $$->pubobj = $1;
> >                                                 $$->location = @1;
> >                                         };
> >
> > You need a single object name under TABLE, not a list -- this was Tom's
> > point about needing post-processing to determine how to assign a type to
> > a object that's what I named PUBLICATIONOBJ_CONTINUATION here.
>
> In the above, we will not be able to use qualified_name, as
> qualified_name will not support the following syntaxes:
> create publication pub1 for table t1 *;
> create publication pub1 for table ONLY t1 *;
> create publication pub1 for table ONLY (t1);
>
> To solve this problem we can change qualified_name to relation_expr
> but the problem with doing that is that the user will be able to
> provide the following syntaxes:
> create publication pub1 for all tables in schema sch1 *;
> create publication pub1 for all tables in schema ONLY sch1 *;
> create publication pub1 for all tables in schema ONLY (sch1);
>
> To handle this we will need some special flag which will differentiate
> these and throw errors at post processing time. We need to define an
> expression similar to relation_expr say pub_expr which handles all
> variants of qualified_name and then use a special flag so that we can
> throw an error if somebody uses the above type of syntax for schema
> names. And then if we have to distinguish between schema name and
> relation name variant, then we need few other things.
>
> We proposed the below solution which handles all these problems and
> also used Node type which need not store schemaname in RangeVar type:
>

Alvaro, do you have any thoughts on these proposed grammar changes?

-- 
With Regards,
Amit Kapila.

  56. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2021-09-23T19:15:12Z

    Hi,

I wanted to do a review of this patch, but I'm a bit confused about 
which patch(es) to review. There's the v5 patch, and then these two 
patches - which seem to be somewhat duplicate, though.

Can anyone explain what's the "current" patch version, or perhaps tell 
me which of the patches to combine?


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  57. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-24T03:10:30Z

    On Fri, Sep 24, 2021 at 12:45 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> Hi,
>
> I wanted to do a review of this patch, but I'm a bit confused about
> which patch(es) to review. There's the v5 patch, and then these two
> patches - which seem to be somewhat duplicate, though.
>
> Can anyone explain what's the "current" patch version, or perhaps tell
> me which of the patches to combine?
>

I think v5 won't work atop a common grammar patch. There need some
adjustments in v5. I think it would be good if we can first get the
common grammar patch reviewed/committed and then build this on top of
it. The common grammar and the corresponding implementation are being
accomplished in the Schema support patch, the latest version of which
is at [1]. Now, Vignesh seems to have extracted just the grammar
portion of that work in his patch
Generic_object_type_parser_002_table_schema_publication [2] (there are
some changes after that but not anything fundamentally different till
now) then he seems to have prepared a patch
(Generic_object_type_parser_001_table_publication [2]) on similar
lines only for tables.

[1] - https://www.postgresql.org/message-id/OS3PR01MB571844A87B6A83B7C10F9D6B94A39%40OS3PR01MB5718.jpnprd01.prod.outlook.com
[2] - https://www.postgresql.org/message-id/CALDaNm1YoxJCs%3DuiyPM%3DtFDDc2qn0ja01nb2TCPqrjZH2jR0sQ%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  58. Re: Column Filtering in Logical Replication

    vignesh C <vignesh21@gmail.com> — 2021-09-24T05:05:29Z

    On Fri, Sep 24, 2021 at 8:40 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Sep 24, 2021 at 12:45 AM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
> >
> > Hi,
> >
> > I wanted to do a review of this patch, but I'm a bit confused about
> > which patch(es) to review. There's the v5 patch, and then these two
> > patches - which seem to be somewhat duplicate, though.
> >
> > Can anyone explain what's the "current" patch version, or perhaps tell
> > me which of the patches to combine?
> >
>
> I think v5 won't work atop a common grammar patch. There need some
> adjustments in v5. I think it would be good if we can first get the
> common grammar patch reviewed/committed and then build this on top of
> it. The common grammar and the corresponding implementation are being
> accomplished in the Schema support patch, the latest version of which
> is at [1].

I have posted an updated patch with the fixes at [1], please review
the updated patch.
[1] - https://www.postgresql.org/message-id/CALDaNm1R-xbQvz4LU5OXu3KKwbWOz3uDcT_YjRU6V0R5FZDYDg%40mail.gmail.com

Regards,
Vignesh

  59. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-24T13:25:09Z

    On 2021-Sep-23, Amit Kapila wrote:

> Alvaro, do you have any thoughts on these proposed grammar changes?

Yeah, I think pubobj_name remains a problem in that you don't know its
return type -- could be a String or a RangeVar, and the user of that
production can't distinguish.  So you're still (unnecessarily, IMV)
stashing an object of undetermined type into ->object.

I think you should get rid of both pubobj_name and pubobj_expr and do
somethine like this:

/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
PublicationObjSpec:	TABLE ColId
					{
						$$ = makeNode(PublicationObjSpec);
						$$->pubobjtype = PUBLICATIONOBJ_TABLE;
						$$->rangevar = makeRangeVarFromQualifiedName($1, NULL, @1, yyscanner);
					}
			| TABLE ColId indirection
					{
						$$ = makeNode(PublicationObjSpec);
						$$->pubobjtype = PUBLICATIONOBJ_TABLE;
						$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
					}
			| ALL TABLES IN_P SCHEMA ColId
					{
						$$ = makeNode(PublicationObjSpec);
						$$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
						$$->name = $4;
					}
			| ALL TABLES IN_P SCHEMA CURRENT_SCHEMA	/* XXX should this be "IN_P CURRENT_SCHEMA"? */
					{
						$$ = makeNode(PublicationObjSpec);
						$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
						$$->name = $4;
					}
			| ColId
					{
						$$ = makeNode(PublicationObjSpec);
						$$->name = $1;
						$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
					}
			| ColId indirection
					{
						$$ = makeNode(PublicationObjSpec);
						$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
						$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
					}
			| CURRENT_SCHEMA
					{
						$$ = makeNode(PublicationObjSpec);
						$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
					}
		;

so in AlterPublicationStmt you would have stanzas like

			| ALTER PUBLICATION name ADD_P pub_obj_list
				{
					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
					n->pubname = $3;
					n->pubobjects = preprocess_pubobj_list($5);
					n->action = DEFELEM_ADD;
					$$ = (Node *)n;
				}

where preprocess_pubobj_list (defined right after processCASbits and
somewhat mimicking it and SplitColQualList) takes all
PUBLICATIONOBJ_CONTINUATION and turns them into either
PUBLICATIONOBJ_TABLE entries or PUBLICATIONOBJ_REL_IN_SCHEMA entries,
depending on what the previous entry was.  (And of course if there is no
previous entry, raise an error immediately).  Note that node
PublicationObjSpec now has two fields, one for RangeVar and another for
a plain name, and tables always use the second one, except when they are
continuations, but of course those continuations that use name are
turned into rangevars in the preprocess step.  I think that would make
the code in ObjectsInPublicationToOids less messy.

(I don't think using the string "CURRENT_SCHEMA" is a great solution.
Did you try having a schema named CURRENT_SCHEMA?)

I verified that bison is happy with the grammar I proposed; I also
verified that you can add opt_column_list to the stanzas for tables, and
it remains happy.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
Y una voz del caos me habló y me dijo
"Sonríe y sé feliz, podría ser peor".
Y sonreí. Y fui feliz.
Y fue peor.

  60. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2021-09-24T20:54:40Z

    
On 9/24/21 7:05 AM, vignesh C wrote:
> On Fri, Sep 24, 2021 at 8:40 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Fri, Sep 24, 2021 at 12:45 AM Tomas Vondra
>> <tomas.vondra@enterprisedb.com> wrote:
>>>
>>> Hi,
>>>
>>> I wanted to do a review of this patch, but I'm a bit confused about
>>> which patch(es) to review. There's the v5 patch, and then these two
>>> patches - which seem to be somewhat duplicate, though.
>>>
>>> Can anyone explain what's the "current" patch version, or perhaps tell
>>> me which of the patches to combine?
>>>
>>
>> I think v5 won't work atop a common grammar patch. There need some
>> adjustments in v5. I think it would be good if we can first get the
>> common grammar patch reviewed/committed and then build this on top of
>> it. The common grammar and the corresponding implementation are being
>> accomplished in the Schema support patch, the latest version of which
>> is at [1].
> 
> I have posted an updated patch with the fixes at [1], please review
> the updated patch.
> [1] - https://www.postgresql.org/message-id/CALDaNm1R-xbQvz4LU5OXu3KKwbWOz3uDcT_YjRU6V0R5FZDYDg%40mail.gmail.com
> 

But that's not the column filtering patch, right? Why would this patch 
depend on "schema level support", but maybe the consensus is there's 
some common part that we need to get in first?

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  61. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-24T22:24:19Z

    On 2021-Sep-24, Tomas Vondra wrote:

> But that's not the column filtering patch, right? Why would this patch
> depend on "schema level support", but maybe the consensus is there's some
> common part that we need to get in first?

Yes, the grammar needs to be common.  I posted a proposed grammar in
https://www.postgresql.org/message-id/202109241325.eag5g6mpvoup%40alvherre.pgsql
(this thread) which should serve both.  I forgot to test the addition of
a WHERE clause for row filtering, though, and I didn't think to look at
adding SEQUENCE support either.

(I'm not sure what's going to be the proposal regarding FOR ALL TABLES
IN SCHEMA for sequences.  Are we going to have "FOR ALL SEQUENCES IN
SCHEMA" and "FOR ALL TABLES AND SEQUENCES IN SCHEMA"?)

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
Thou shalt study thy libraries and strive not to reinvent them without
cause, that thy code may be short and readable and thy days pleasant
and productive. (7th Commandment for C Programmers)

  62. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2021-09-24T22:30:12Z

    On 9/25/21 12:24 AM, Alvaro Herrera wrote:
> On 2021-Sep-24, Tomas Vondra wrote:
> 
>> But that's not the column filtering patch, right? Why would this patch
>> depend on "schema level support", but maybe the consensus is there's some
>> common part that we need to get in first?
> 
> Yes, the grammar needs to be common.  I posted a proposed grammar in
> https://www.postgresql.org/message-id/202109241325.eag5g6mpvoup%40alvherre.pgsql
> (this thread) which should serve both.  I forgot to test the addition of
> a WHERE clause for row filtering, though, and I didn't think to look at
> adding SEQUENCE support either.
> 

Fine with me, but I still don't know which version of the column 
filtering patch should I look at ... maybe there's none up to date, at 
the moment?

> (I'm not sure what's going to be the proposal regarding FOR ALL TABLES
> IN SCHEMA for sequences.  Are we going to have "FOR ALL SEQUENCES IN
> SCHEMA" and "FOR ALL TABLES AND SEQUENCES IN SCHEMA"?)
> 

Should be "FOR ABSOLUTELY EVERYTHING IN SCHEMA" of course ;-)

On a more serious note, a comma-separated list of objects seems like the 
best / most flexible choice, i.e. "FOR TABLES, SEQUENCES IN SCHEMA"?


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  63. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-24T23:00:47Z

    On 2021-Sep-25, Tomas Vondra wrote:

> On 9/25/21 12:24 AM, Alvaro Herrera wrote:
> > On 2021-Sep-24, Tomas Vondra wrote:
> > 
> > > But that's not the column filtering patch, right? Why would this patch
> > > depend on "schema level support", but maybe the consensus is there's some
> > > common part that we need to get in first?
> > 
> > Yes, the grammar needs to be common.  I posted a proposed grammar in
> > https://www.postgresql.org/message-id/202109241325.eag5g6mpvoup%40alvherre.pgsql
> > (this thread) which should serve both.  I forgot to test the addition of
> > a WHERE clause for row filtering, though, and I didn't think to look at
> > adding SEQUENCE support either.
> 
> Fine with me, but I still don't know which version of the column filtering
> patch should I look at ... maybe there's none up to date, at the moment?

I don't think there is one.  I think the latest is what I posted in
https://postgr.es/m/202109061751.3qz5xpugwx6w@alvherre.pgsql (At least I
don't see any reply from Rahila with attachments after that), but that
wasn't addressing a bunch of review comments that had been made; and I
suspect that Amit K has already committed a few conflicting patches
after that.

> > (I'm not sure what's going to be the proposal regarding FOR ALL TABLES
> > IN SCHEMA for sequences.  Are we going to have "FOR ALL SEQUENCES IN
> > SCHEMA" and "FOR ALL TABLES AND SEQUENCES IN SCHEMA"?)
> 
> Should be "FOR ABSOLUTELY EVERYTHING IN SCHEMA" of course ;-)

hahah ...

> On a more serious note, a comma-separated list of objects seems like the
> best / most flexible choice, i.e. "FOR TABLES, SEQUENCES IN SCHEMA"?

Hmm, not sure if bison is going to like that.  Maybe it's OK if
SEQUENCES is a fully reserved word?  But nothing beats experimentation!

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
Thou shalt check the array bounds of all strings (indeed, all arrays), for
surely where thou typest "foo" someone someday shall type
"supercalifragilisticexpialidocious" (5th Commandment for C programmers)

  64. RE: Column Filtering in Logical Replication

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2021-09-25T01:54:12Z

    From Fri, Sep 24, 2021 9:25 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> On 2021-Sep-23, Amit Kapila wrote:
> 
> > Alvaro, do you have any thoughts on these proposed grammar changes?
> 
> Yeah, I think pubobj_name remains a problem in that you don't know its return
> type -- could be a String or a RangeVar, and the user of that production can't
> distinguish.  So you're still (unnecessarily, IMV) stashing an object of
> undetermined type into ->object.
> 
> I think you should get rid of both pubobj_name and pubobj_expr and do
> somethine like this:
> PublicationObjSpec:	TABLE ColId
> 					{
> 						$$ = makeNode(PublicationObjSpec);
> 						$$->pubobjtype = PUBLICATIONOBJ_TABLE;
> 						$$->rangevar = makeRangeVarFromQualifiedName($1, NULL, @1, yyscanner);
> 					}
> 			| TABLE ColId indirection
> 					{
> 						$$ = makeNode(PublicationObjSpec);
> 						$$->pubobjtype = PUBLICATIONOBJ_TABLE;
> 						$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
> 					}

Hi,

IIRC, the above grammar doesn't support extended relation expression (like:
"tablename * ", "ONLY tablename", "ONLY '( tablename )") which is part of rule
relation_expr. I think we should add these too. And if we move forward with the
design you proposed, we should do something like the following:

/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
PublicationObjSpec:
			TABLE relation_expr
							{
								$$ = makeNode(PublicationObjSpec);
								$$->pubobjtype = PUBLICATIONOBJ_TABLE;
								$$->rangevar = $2;
							}
			| ALL TABLES IN_P SCHEMA ColId
							{
								$$ = makeNode(PublicationObjSpec);
								$$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
								$$->name = $5;
							}
			| ALL TABLES IN_P SCHEMA CURRENT_SCHEMA
							{
								$$ = makeNode(PublicationObjSpec);
								$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
								$$->name = $5;
							}
			| extended_relation_expr	/* grammar like tablename * , ONLY tablename, ONLY ( tablename )*/
							{
								$$ = makeNode(PublicationObjSpec);
								$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
								$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
							}
			| ColId
							{
								$$ = makeNode(PublicationObjSpec);
								$$->name = $1;
								$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;

							}
			| ColId indirection
							{
								$$ = makeNode(PublicationObjSpec);
								$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
								$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;

							}

Best regards,
Hou zj

  65. Re: Column Filtering in Logical Replication

    vignesh C <vignesh21@gmail.com> — 2021-09-25T07:45:08Z

    On Fri, Sep 24, 2021 at 6:55 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-23, Amit Kapila wrote:
>
> > Alvaro, do you have any thoughts on these proposed grammar changes?
>
> Yeah, I think pubobj_name remains a problem in that you don't know its
> return type -- could be a String or a RangeVar, and the user of that
> production can't distinguish.  So you're still (unnecessarily, IMV)
> stashing an object of undetermined type into ->object.
>
> I think you should get rid of both pubobj_name and pubobj_expr and do
> somethine like this:
>
> /* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
> PublicationObjSpec:     TABLE ColId
>                                         {
>                                                 $$ = makeNode(PublicationObjSpec);
>                                                 $$->pubobjtype = PUBLICATIONOBJ_TABLE;
>                                                 $$->rangevar = makeRangeVarFromQualifiedName($1, NULL, @1, yyscanner);
>                                         }
>                         | TABLE ColId indirection
>                                         {
>                                                 $$ = makeNode(PublicationObjSpec);
>                                                 $$->pubobjtype = PUBLICATIONOBJ_TABLE;
>                                                 $$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
>                                         }
>                         | ALL TABLES IN_P SCHEMA ColId
>                                         {
>                                                 $$ = makeNode(PublicationObjSpec);
>                                                 $$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
>                                                 $$->name = $4;
>                                         }
>                         | ALL TABLES IN_P SCHEMA CURRENT_SCHEMA /* XXX should this be "IN_P CURRENT_SCHEMA"? */
>                                         {
>                                                 $$ = makeNode(PublicationObjSpec);
>                                                 $$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
>                                                 $$->name = $4;
>                                         }
>                         | ColId
>                                         {
>                                                 $$ = makeNode(PublicationObjSpec);
>                                                 $$->name = $1;
>                                                 $$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
>                                         }
>                         | ColId indirection
>                                         {
>                                                 $$ = makeNode(PublicationObjSpec);
>                                                 $$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
>                                                 $$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
>                                         }
>                         | CURRENT_SCHEMA
>                                         {
>                                                 $$ = makeNode(PublicationObjSpec);
>                                                 $$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
>                                         }
>                 ;

Apart from the issue that Hou San pointed, I found one issue with
introduction of PUBLICATIONOBJ_CURRSCHEMA, I was not able to
differentiate if it is table or schema in the following cases:
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA CURRENT_SCHEMA;
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA sch1, CURRENT_SCHEMA;
CREATE PUBLICATION pub1 FOR table t1, CURRENT_SCHEMA;
The differentiation is required to differentiate and add a schema or a table.

I felt it was better to use PUBLICATIONOBJ_CONTINUATION in case of
CURRENT_SCHEMA in multiple object cases like:
PublicationObjSpec:     TABLE relation_expr
                                        {
                                                $$ =
makeNode(PublicationObjSpec);
                                                $$->pubobjtype =
PUBLICATIONOBJ_TABLE;
                                                $$->rangevar = $2;
                                        }
                        | ALL TABLES IN_P SCHEMA ColId
                                        {
                                                $$ =
makeNode(PublicationObjSpec);
                                                $$->pubobjtype =
PUBLICATIONOBJ_REL_IN_SCHEMA;
                                                $$->name = $5;
                                                $$->location = @5;
                                        }
                        | ALL TABLES IN_P SCHEMA CURRENT_SCHEMA /* XXX
should this be "IN_P CURRENT_SCHEMA"? */
                                        {
                                                $$ =
makeNode(PublicationObjSpec);
                                                $$->pubobjtype =
PUBLICATIONOBJ_REL_IN_SCHEMA;
                                                $$->name = "CURRENT_SCHEMA";
                                                $$->location = @5;
                                        }
                        | ColId
                                        {
                                                $$ =
makeNode(PublicationObjSpec);
                                                $$->name = $1;
                                                $$->pubobjtype =
PUBLICATIONOBJ_CONTINUATION;
                                                $$->location = @1;
                                        }
                        | ColId indirection
                                        {
                                                $$ =
makeNode(PublicationObjSpec);
                                                $$->rangevar =
makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
                                                $$->pubobjtype =
PUBLICATIONOBJ_CONTINUATION;
                                                $$->location = @1;
                                        }
                        | CURRENT_SCHEMA
                                        {
                                                $$ =
makeNode(PublicationObjSpec);
                                                $$->pubobjtype =
PUBLICATIONOBJ_CONTINUATION;
                                                $$->name = "CURRENT_SCHEMA";
                                                $$->location = @1;
                                        }
                        | extended_relation_expr        /* grammar
like tablename * , ONLY tablename, ONLY ( tablename )*/
                                        {
                                                $$ =
makeNode(PublicationObjSpec);
                                                /*$$->rangevar =
makeRangeVarFromQualifiedName($1, $2, @1, yyscanner); */
                                                $$->rangevar = $1;
                                                $$->pubobjtype =
PUBLICATIONOBJ_CONTINUATION;
                                        }
                                ;

I'm ok with your suggestion along with the above proposed changes. I
felt the changes proposed at [1] were also fine. Let's change it to
whichever is better, easily extendable and can handle the Column
filtering project, ALL TABLES IN SCHEMA, ALL SEQUENCES IN SCHEMA
projects, and other projects in the future. Based on that we can check
in the parser changes independently and then the remaining series of
the patches can be rebased on top of it accordingly. Thoughts?

> so in AlterPublicationStmt you would have stanzas like
>
>                         | ALTER PUBLICATION name ADD_P pub_obj_list
>                                 {
>                                         AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
>                                         n->pubname = $3;
>                                         n->pubobjects = preprocess_pubobj_list($5);
>                                         n->action = DEFELEM_ADD;
>                                         $$ = (Node *)n;
>                                 }
>
> where preprocess_pubobj_list (defined right after processCASbits and
> somewhat mimicking it and SplitColQualList) takes all
> PUBLICATIONOBJ_CONTINUATION and turns them into either
> PUBLICATIONOBJ_TABLE entries or PUBLICATIONOBJ_REL_IN_SCHEMA entries,
> depending on what the previous entry was.  (And of course if there is no
> previous entry, raise an error immediately).  Note that node
> PublicationObjSpec now has two fields, one for RangeVar and another for
> a plain name, and tables always use the second one, except when they are
> continuations, but of course those continuations that use name are
> turned into rangevars in the preprocess step.  I think that would make
> the code in ObjectsInPublicationToOids less messy.

I agree with this. I will make the changes for this in the next version.

> (I don't think using the string "CURRENT_SCHEMA" is a great solution.
> Did you try having a schema named CURRENT_SCHEMA?)

Here CURRENT_SCHEMA is not used for the schema name, it will be
replaced with the name of the schema that is first in the search path.

[1] - https://www.postgresql.org/message-id/CALDaNm1R-xbQvz4LU5OXu3KKwbWOz3uDcT_YjRU6V0R5FZDYDg%40mail.gmail.com

Regards,
Vignesh

  66. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-27T11:10:50Z

    On Sat, Sep 25, 2021 at 1:15 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, Sep 24, 2021 at 6:55 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> >
> > On 2021-Sep-23, Amit Kapila wrote:
> >
> > > Alvaro, do you have any thoughts on these proposed grammar changes?
> >
> > Yeah, I think pubobj_name remains a problem in that you don't know its
> > return type -- could be a String or a RangeVar, and the user of that
> > production can't distinguish.  So you're still (unnecessarily, IMV)
> > stashing an object of undetermined type into ->object.
> >
> > I think you should get rid of both pubobj_name and pubobj_expr and do
> > somethine like this:
> >
> > /* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
> > PublicationObjSpec:     TABLE ColId
> >                                         {
> >                                                 $$ = makeNode(PublicationObjSpec);
> >                                                 $$->pubobjtype = PUBLICATIONOBJ_TABLE;
> >                                                 $$->rangevar = makeRangeVarFromQualifiedName($1, NULL, @1, yyscanner);
> >                                         }
> >                         | TABLE ColId indirection
> >                                         {
> >                                                 $$ = makeNode(PublicationObjSpec);
> >                                                 $$->pubobjtype = PUBLICATIONOBJ_TABLE;
> >                                                 $$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
> >                                         }
> >                         | ALL TABLES IN_P SCHEMA ColId
> >                                         {
> >                                                 $$ = makeNode(PublicationObjSpec);
> >                                                 $$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
> >                                                 $$->name = $4;
> >                                         }
> >                         | ALL TABLES IN_P SCHEMA CURRENT_SCHEMA /* XXX should this be "IN_P CURRENT_SCHEMA"? */
> >                                         {
> >                                                 $$ = makeNode(PublicationObjSpec);
> >                                                 $$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
> >                                                 $$->name = $4;
> >                                         }
> >                         | ColId
> >                                         {
> >                                                 $$ = makeNode(PublicationObjSpec);
> >                                                 $$->name = $1;
> >                                                 $$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
> >                                         }
> >                         | ColId indirection
> >                                         {
> >                                                 $$ = makeNode(PublicationObjSpec);
> >                                                 $$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
> >                                                 $$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
> >                                         }
> >                         | CURRENT_SCHEMA
> >                                         {
> >                                                 $$ = makeNode(PublicationObjSpec);
> >                                                 $$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
> >                                         }
> >                 ;
>
> Apart from the issue that Hou San pointed, I found one issue with
> introduction of PUBLICATIONOBJ_CURRSCHEMA, I was not able to
> differentiate if it is table or schema in the following cases:
> CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA CURRENT_SCHEMA;
> CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA sch1, CURRENT_SCHEMA;
> CREATE PUBLICATION pub1 FOR table t1, CURRENT_SCHEMA;
> The differentiation is required to differentiate and add a schema or a table.
>

I am not sure what makes you say that we can't distinguish the above
cases when there is already a separate rule for CURRENT_SCHEMA? I
think you can distinguish by tracking the previous objects as we are
already doing in the patch. But one thing that is not clear to me is
is the reason to introduce a new type PUBLICATIONOBJ_CURRSCHEMA when
we use PUBLICATIONOBJ_REL_IN_SCHEMA and PUBLICATIONOBJ_CONTINUATION to
distinguish all cases of CURRENT_SCHEMA. Alvaro might have something
in mind for this which is not apparent and that might have caused
confusion to you as well?

-- 
With Regards,
Amit Kapila.

  67. Re: Column Filtering in Logical Replication

    vignesh C <vignesh21@gmail.com> — 2021-09-27T11:43:55Z

    On Mon, Sep 27, 2021 at 4:41 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sat, Sep 25, 2021 at 1:15 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Fri, Sep 24, 2021 at 6:55 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> > >
> > > On 2021-Sep-23, Amit Kapila wrote:
> > >
> > > > Alvaro, do you have any thoughts on these proposed grammar changes?
> > >
> > > Yeah, I think pubobj_name remains a problem in that you don't know its
> > > return type -- could be a String or a RangeVar, and the user of that
> > > production can't distinguish.  So you're still (unnecessarily, IMV)
> > > stashing an object of undetermined type into ->object.
> > >
> > > I think you should get rid of both pubobj_name and pubobj_expr and do
> > > somethine like this:
> > >
> > > /* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
> > > PublicationObjSpec:     TABLE ColId
> > >                                         {
> > >                                                 $$ = makeNode(PublicationObjSpec);
> > >                                                 $$->pubobjtype = PUBLICATIONOBJ_TABLE;
> > >                                                 $$->rangevar = makeRangeVarFromQualifiedName($1, NULL, @1, yyscanner);
> > >                                         }
> > >                         | TABLE ColId indirection
> > >                                         {
> > >                                                 $$ = makeNode(PublicationObjSpec);
> > >                                                 $$->pubobjtype = PUBLICATIONOBJ_TABLE;
> > >                                                 $$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
> > >                                         }
> > >                         | ALL TABLES IN_P SCHEMA ColId
> > >                                         {
> > >                                                 $$ = makeNode(PublicationObjSpec);
> > >                                                 $$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
> > >                                                 $$->name = $4;
> > >                                         }
> > >                         | ALL TABLES IN_P SCHEMA CURRENT_SCHEMA /* XXX should this be "IN_P CURRENT_SCHEMA"? */
> > >                                         {
> > >                                                 $$ = makeNode(PublicationObjSpec);
> > >                                                 $$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
> > >                                                 $$->name = $4;
> > >                                         }
> > >                         | ColId
> > >                                         {
> > >                                                 $$ = makeNode(PublicationObjSpec);
> > >                                                 $$->name = $1;
> > >                                                 $$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
> > >                                         }
> > >                         | ColId indirection
> > >                                         {
> > >                                                 $$ = makeNode(PublicationObjSpec);
> > >                                                 $$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
> > >                                                 $$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
> > >                                         }
> > >                         | CURRENT_SCHEMA
> > >                                         {
> > >                                                 $$ = makeNode(PublicationObjSpec);
> > >                                                 $$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
> > >                                         }
> > >                 ;
> >
> > Apart from the issue that Hou San pointed, I found one issue with
> > introduction of PUBLICATIONOBJ_CURRSCHEMA, I was not able to
> > differentiate if it is table or schema in the following cases:
> > CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA CURRENT_SCHEMA;
> > CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA sch1, CURRENT_SCHEMA;
> > CREATE PUBLICATION pub1 FOR table t1, CURRENT_SCHEMA;
> > The differentiation is required to differentiate and add a schema or a table.
> >
>
> I am not sure what makes you say that we can't distinguish the above
> cases when there is already a separate rule for CURRENT_SCHEMA? I
> think you can distinguish by tracking the previous objects as we are
> already doing in the patch. But one thing that is not clear to me is
> is the reason to introduce a new type PUBLICATIONOBJ_CURRSCHEMA when
> we use PUBLICATIONOBJ_REL_IN_SCHEMA and PUBLICATIONOBJ_CONTINUATION to
> distinguish all cases of CURRENT_SCHEMA. Alvaro might have something
> in mind for this which is not apparent and that might have caused
> confusion to you as well?

It is difficult to identify this case:
1) create publication pub1 for all tables in schema CURRENT_SCHEMA;
2) create publication pub1 for CURRENT_SCHEMA;

Here case 1 should succeed and case 2 should throw error:
Since the object type will be set to PUBLICATIONOBJ_CURRSCHEMA in both
cases, we cannot differentiate between them:
1) ALL TABLES IN_P SCHEMA CURRENT_SCHEMA /* XXX should this be "IN_P
CURRENT_SCHEMA"? */
                                        {
                                                $$ =
makeNode(PublicationObjSpec);
                                                $$->pubobjtype =
PUBLICATIONOBJ_CURRSCHEMA;
                                                $$->name = $4;
                                        }
2) CURRENT_SCHEMA
                                        {
                                                $$ =
makeNode(PublicationObjSpec);
                                                $$->pubobjtype =
PUBLICATIONOBJ_CURRSCHEMA;
                                        }

I felt it will work, if we set object type to
PUBLICATIONOBJ_CONTINUATION in 2nd case(CURRENT_SCHEMA) and setting
object type to PUBLICATIONOBJ_REL_IN_SCHEMA or
PUBLICATIONOBJ_CURRSCHEMA in 1st case( ALL TABLES IN_P SCHEMA
CURRENT_SCHEMA).
Thoughts?

Regards,
Vignesh

  68. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-27T12:23:49Z

    On 2021-Sep-27, Amit Kapila wrote:

> I am not sure what makes you say that we can't distinguish the above
> cases when there is already a separate rule for CURRENT_SCHEMA? I
> think you can distinguish by tracking the previous objects as we are
> already doing in the patch. But one thing that is not clear to me is
> is the reason to introduce a new type PUBLICATIONOBJ_CURRSCHEMA when
> we use PUBLICATIONOBJ_REL_IN_SCHEMA and PUBLICATIONOBJ_CONTINUATION to
> distinguish all cases of CURRENT_SCHEMA. Alvaro might have something
> in mind for this which is not apparent and that might have caused
> confusion to you as well?

My issue is what happens if you have a schema that is named
CURRENT_SCHEMA.  In the normal case where you do ALL TABLES IN SCHEMA
"CURRENT_SCHEMA" you would end up with a String containing
"CURRENT_SCHEMA", so how do you distinguish that from ALL TABLES IN
SCHEMA CURRENT_SCHEMA, which does not refer to the schema named
"CURRENT_SCHEMA" but in Vignesh's proposal also uses a String containing
"CURRENT_SCHEMA"?

Now you could say "but who would be stupid enough to do that??!", but it
seems easier to dodge the problem entirely.  AFAICS our grammar never
uses String "CURRENT_SCHEMA" to represent CURRENT_SCHEMA, but rather
some special enum value.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/

  69. Re: Column Filtering in Logical Replication

    Rahila Syed <rahilasyed90@gmail.com> — 2021-09-27T13:10:57Z

    Hi,


>
> I don't think there is one.  I think the latest is what I posted in
> https://postgr.es/m/202109061751.3qz5xpugwx6w@alvherre.pgsql (At least I
> don't see any reply from Rahila with attachments after that), but that
> wasn't addressing a bunch of review comments that had been made; and I
> suspect that Amit K has already committed a few conflicting patches
> after that.
>
> Yes, the v5 version of the patch attached by Alvaro is the latest one.
IIUC, the review comments that are yet to be addressed apart from the
ongoing grammar
discussion, are as follows:

1. Behaviour on dropping a column from the table, that is a part of column
filter.
In the latest patch, the entire table is dropped from publication on
dropping a column
that is a part of the column filter. However, there is preference for
another approach
to drop just the column from the filter on DROP column CASCADE(continue to
filter
the other columns), and an error for DROP RESTRICT.

2. Instead of WITH RECURSIVE query to find the topmost parent of the
partition
in fetch_remote_table_info, use pg_partition_tree and pg_partition_root.

3. Report of memory leakage in get_rel_sync_entry().

4. Missing documentation

5. Latest comments(last two messages) by Peter Smith.

Thank you,
Rahila Syed

  70. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-28T03:29:51Z

    On Mon, Sep 27, 2021 at 5:53 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-27, Amit Kapila wrote:
>
> > I am not sure what makes you say that we can't distinguish the above
> > cases when there is already a separate rule for CURRENT_SCHEMA? I
> > think you can distinguish by tracking the previous objects as we are
> > already doing in the patch. But one thing that is not clear to me is
> > is the reason to introduce a new type PUBLICATIONOBJ_CURRSCHEMA when
> > we use PUBLICATIONOBJ_REL_IN_SCHEMA and PUBLICATIONOBJ_CONTINUATION to
> > distinguish all cases of CURRENT_SCHEMA. Alvaro might have something
> > in mind for this which is not apparent and that might have caused
> > confusion to you as well?
>
> My issue is what happens if you have a schema that is named
> CURRENT_SCHEMA.  In the normal case where you do ALL TABLES IN SCHEMA
> "CURRENT_SCHEMA" you would end up with a String containing
> "CURRENT_SCHEMA", so how do you distinguish that from ALL TABLES IN
> SCHEMA CURRENT_SCHEMA, which does not refer to the schema named
> "CURRENT_SCHEMA" but in Vignesh's proposal also uses a String containing
> "CURRENT_SCHEMA"?
>
> Now you could say "but who would be stupid enough to do that??!",
>

But it is not allowed to create schema or table with the name
CURRENT_SCHEMA, so not sure if we need to do anything special for it.
However, if we want to handle it as a separate enum then the handling
would be something like:

| ALL TABLES IN_P SCHEMA CURRENT_SCHEMA
                                        {
                                                $$ =
makeNode(PublicationObjSpec);
                                                $$->pubobjtype =
PUBLICATIONOBJ_CURRSCHEMA;
                                        }
...
...
| CURRENT_SCHEMA
                                        {
                                                $$ =
makeNode(PublicationObjSpec);
                                                $$->pubobjtype =
PUBLICATIONOBJ_CONTINUATION;
                                        }
                ;

Now, during post-processing, the PUBLICATIONOBJ_CONTINUATION will be
distinguished as CURRENT_SCHEMA because both rangeVar and name will be
NULL. Do you have other ideas to deal with it? Vignesh has already
point in his email [1] why we can't keep pubobjtype as
PUBLICATIONOBJ_CURRSCHEMA in the second case, so I used
PUBLICATIONOBJ_CONTINUATION.

[1] - https://www.postgresql.org/message-id/CALDaNm06shp%2BALwC2s-dV-S4k2o6bcmXnXGX4ETkoXxKHQfjfA%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  71. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-28T04:01:33Z

    On Mon, Sep 27, 2021 at 6:41 PM Rahila Syed <rahilasyed90@gmail.com> wrote:
>
>>
>>
>> I don't think there is one.  I think the latest is what I posted in
>> https://postgr.es/m/202109061751.3qz5xpugwx6w@alvherre.pgsql (At least I
>> don't see any reply from Rahila with attachments after that), but that
>> wasn't addressing a bunch of review comments that had been made; and I
>> suspect that Amit K has already committed a few conflicting patches
>> after that.
>>
> Yes, the v5 version of the patch attached by Alvaro is the latest one.
> IIUC, the review comments that are yet to be addressed apart from the ongoing grammar
> discussion, are as follows:
>
> 1. Behaviour on dropping a column from the table, that is a part of column filter.
> In the latest patch, the entire table is dropped from publication on dropping a column
> that is a part of the column filter. However, there is preference for another approach
> to drop just the column from the filter on DROP column CASCADE(continue to filter
> the other columns), and an error for DROP RESTRICT.
>

I am not sure if we can do this as pointed by me in one of the
previous emails [1]. I think additionally, you might want to take some
action if the replica identity is changed as requested in the same
email [1].

[1] - https://www.postgresql.org/message-id/CAA4eK1KCGF43pfLv8%2BmixcTMs%3DNkd6YdWL53LhiT1DvnuTg01g%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  72. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-09-29T13:19:35Z

    On 2021-Sep-28, Amit Kapila wrote:

> But it is not allowed to create schema or table with the name
> CURRENT_SCHEMA, so not sure if we need to do anything special for it.

Oh?  You certainly can.

alvherre=# create schema "CURRENT_SCHEMA";
CREATE SCHEMA
alvherre=# \dn
        Listado de esquemas
     Nombre     |       Dueño       
----------------+-------------------
 CURRENT_SCHEMA | alvherre
 public         | pg_database_owner
 temp           | alvherre
(3 filas)

alvherre=# create table "CURRENT_SCHEMA"."CURRENT_SCHEMA" ("bother amit for a while" int);
CREATE TABLE
alvherre=# \d "CURRENT_SCHEMA".*
                  Tabla «CURRENT_SCHEMA.CURRENT_SCHEMA»
         Columna         |  Tipo   | Ordenamiento | Nulable | Por omisión 
-------------------------+---------+--------------+---------+-------------
 bother amit for a while | integer |              |         | 


> Now, during post-processing, the PUBLICATIONOBJ_CONTINUATION will be
> distinguished as CURRENT_SCHEMA because both rangeVar and name will be
> NULL. Do you have other ideas to deal with it?

That sounds plausible.  There's no need for a name-free object of any other
kind AFAICS, so there should be no conflict.  If we ever do find a
conflict, we can add another struct member to disambiguate.

Thanks

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
"Doing what he did amounts to sticking his fingers under the hood of the
implementation; if he gets his fingers burnt, it's his problem."  (Tom Lane)

  73. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-30T11:09:41Z

    On Wed, Sep 29, 2021 at 6:49 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-28, Amit Kapila wrote:
>
> > But it is not allowed to create schema or table with the name
> > CURRENT_SCHEMA, so not sure if we need to do anything special for it.
>
> Oh?  You certainly can.
>
> alvherre=# create schema "CURRENT_SCHEMA";
> CREATE SCHEMA
> alvherre=# \dn
>         Listado de esquemas
>      Nombre     |       Dueño
> ----------------+-------------------
>  CURRENT_SCHEMA | alvherre
>  public         | pg_database_owner
>  temp           | alvherre
> (3 filas)
>
> alvherre=# create table "CURRENT_SCHEMA"."CURRENT_SCHEMA" ("bother amit for a while" int);
> CREATE TABLE
> alvherre=# \d "CURRENT_SCHEMA".*
>                   Tabla «CURRENT_SCHEMA.CURRENT_SCHEMA»
>          Columna         |  Tipo   | Ordenamiento | Nulable | Por omisión
> -------------------------+---------+--------------+---------+-------------
>  bother amit for a while | integer |              |         |
>

oops, I was trying without quotes.

>
> > Now, during post-processing, the PUBLICATIONOBJ_CONTINUATION will be
> > distinguished as CURRENT_SCHEMA because both rangeVar and name will be
> > NULL. Do you have other ideas to deal with it?
>
> That sounds plausible.  There's no need for a name-free object of any other
> kind AFAICS, so there should be no conflict.  If we ever do find a
> conflict, we can add another struct member to disambiguate.
>

Okay, thanks. I feel now we are in agreement on the grammar rules.


-- 
With Regards,
Amit Kapila.

  74. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-01T20:29:47Z

    Hi

I took the latest posted patch, rebased on current sources, fixed the
conflicts, and pgindented.  No further changes.  Here's the result.  All
tests are passing for me.  Some review comments that were posted have
not been addressed yet; I'll look into that soon.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Java is clearly an example of money oriented programming"  (A. Stepanov)

  75. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-01T20:47:07Z

    Oh, I just noticed that for some reason the test file was lost in the
rebase, so those tests I thought I was running ... I wasn't.  And of
course if I put it back, it fails.

More later.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"Crear es tan difícil como ser libre" (Elsa Triolet)

  76. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-02T14:23:58Z

    On 2021-Dec-01, Alvaro Herrera wrote:

> Hi
> 
> I took the latest posted patch, rebased on current sources, fixed the
> conflicts, and pgindented.  No further changes.  Here's the result.  All
> tests are passing for me.  Some review comments that were posted have
> not been addressed yet; I'll look into that soon.

In v7 I have reinstated the test file and fixed the silly problem that
caused it to fail (probably a mistake of mine while rebasing).

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
Maybe there's lots of data loss but the records of data loss are also lost.
(Lincoln Yeoh)

  77. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-02T19:15:03Z

    On 2021-Sep-16, Peter Smith wrote:

> I noticed that the latest v5 no longer includes the TAP test which was
> in the v4 patch.
> 
> (src/test/subscription/t/021_column_filter.pl)
> 
> Was that omission deliberate?

Somehow I not only failed to notice the omission, but also your email
where you told us about it.  I have since posted a version of the patch
that again includes it.

Thanks!

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"No renuncies a nada. No te aferres a nada."

  78. Re: Column Filtering in Logical Replication

    vignesh C <vignesh21@gmail.com> — 2021-12-03T09:17:39Z

    On Fri, Dec 3, 2021 at 12:45 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-16, Peter Smith wrote:
>
> > I noticed that the latest v5 no longer includes the TAP test which was
> > in the v4 patch.
> >
> > (src/test/subscription/t/021_column_filter.pl)
> >
> > Was that omission deliberate?
>
> Somehow I not only failed to notice the omission, but also your email
> where you told us about it.  I have since posted a version of the patch
> that again includes it.

Thanks for the patch, Few comments:
I had a look at the patch, I felt the following should be handled:
1) Dump changes to include the column filters while adding table to
publication in dumpPublicationTable
2) Documentation changes for column filtering in create_publication.sgml
3) describe publication changes to support \dRp command in describePublications
4) I felt we need not allow specifying columns in case of "alter
publication drop table" as currently dropping column filter is not
allowed.
5) We should check if the column specified is present in the table,
currently we are able to specify non existent column for column
filtering
+       foreach(lc, targetrel->columns)
+       {
+               char       *colname;
+
+               colname = strVal(lfirst(lc));
+               target_cols = lappend(target_cols, colname);
+       }
+       check_publication_add_relation(targetrel->relation, target_cols);

Regards,
Vignesh

  79. Re: Column Filtering in Logical Replication

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2021-12-10T13:08:52Z

    On 02.12.21 15:23, Alvaro Herrera wrote:
>> I took the latest posted patch, rebased on current sources, fixed the
>> conflicts, and pgindented.  No further changes.  Here's the result.  All
>> tests are passing for me.  Some review comments that were posted have
>> not been addressed yet; I'll look into that soon.
> 
> In v7 I have reinstated the test file and fixed the silly problem that
> caused it to fail (probably a mistake of mine while rebasing).

I looked through this a bit.  You had said that you are still going to 
integrate past review comments, so I didn't look to deeply before you 
get to that.

Attached are a few fixup patches that you could integrate.

There was no documentation, so I wrote a bit (patch 0001).  It only 
touches the CREATE PUBLICATION and ALTER PUBLICATION pages at the 
moment.  There was no mention in the Logical Replication chapter that 
warranted updating.  Perhaps we should revisit that chapter at the end 
of the release cycle.

DDL tests should be done in src/test/regress/sql/publication.sql rather 
than through TAP tests, to keep it simpler.  I have added a few that I 
came up with (patch 0002).  Note the FIXME marker that it does not 
recognize if the listed columns don't exist.  I removed a now redundant 
test from the TAP test file.  The other error condition test in the TAP 
test file ('publication relation test_part removed') I didn't 
understand: test_part was added with columns (a, b), so why would 
dropping column b remove the whole entry?  Maybe I missed something, or 
this could be explained better.

I was curious what happens when you have different publications with 
different column lists, so I wrote a test for that (patch 0003).  It 
turns out it works, so there is nothing to do, but perhaps the test is 
useful to keep.

The test file 021_column_filter.pl should be renamed to an unused number 
(would be 027 currently).  Also, it contains references to "TRUNCATE", 
where it was presumably copied from.

On the implementation side, I think the added catalog column 
pg_publication_rel.prattrs should be an int2 array, not a text array. 
That would also fix the above problem.  If you have to look up the 
columns at DDL time, then you will notice when they don't exist.

Finally, I suggest not naming this feature "column filter".  I think 
this name arose because of the analogy with the "row filter" feature 
also being developed.  But a filter is normally a dynamic data-driven 
action, which this is not.  Golden Gate calls it in their documentation 
"Selecting Columns", or we could just call it "column list".

  80. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-10T13:23:51Z

    On 2021-Dec-10, Peter Eisentraut wrote:

> I looked through this a bit.  You had said that you are still going to
> integrate past review comments, so I didn't look to deeply before you get to
> that.

Thanks for doing this!  As it happens I've spent the last couple of days
working on some of these details.

> There was no documentation, so I wrote a bit (patch 0001).  It only touches
> the CREATE PUBLICATION and ALTER PUBLICATION pages at the moment.  There was
> no mention in the Logical Replication chapter that warranted updating.
> Perhaps we should revisit that chapter at the end of the release cycle.

Thanks.  I hadn't looked at the docs yet, so I'll definitely take this.

> DDL tests should be done in src/test/regress/sql/publication.sql rather than
> through TAP tests, to keep it simpler.

Yeah, I noticed this too but hadn't done it yet.

> Note the FIXME marker that it does not recognize if the
> listed columns don't exist.

I had fixed this already, so I suppose it should be okay.

> I removed a now redundant test from the TAP
> test file.  The other error condition test in the TAP test file
> ('publication relation test_part removed') I didn't understand: test_part
> was added with columns (a, b), so why would dropping column b remove the
> whole entry?  Maybe I missed something, or this could be explained better.

There was some discussion about it earlier in the thread and I was also
against this proposed behavior.

> I was curious what happens when you have different publications with
> different column lists, so I wrote a test for that (patch 0003).  It turns
> out it works, so there is nothing to do, but perhaps the test is useful to
> keep.

Great, thanks.  Yes, I think it will be.

> On the implementation side, I think the added catalog column
> pg_publication_rel.prattrs should be an int2 array, not a text array.

I already rewrote it to use a int2vector column in pg_publication_rel.
This interacted badly with the previous behavior on dropping columns,
which I have to revisit, but otherwise it seems much better.
(Particularly since we don't need to care about quoting names and such.)

> Finally, I suggest not naming this feature "column filter".  I think this
> name arose because of the analogy with the "row filter" feature also being
> developed.  But a filter is normally a dynamic data-driven action, which
> this is not.  Golden Gate calls it in their documentation "Selecting
> Columns", or we could just call it "column list".

Hmm, I hadn't thought of renaming the feature, but I have to admit that
I was confused because of the name, so I agree with choosing some other
name.

I'll integrate your changes and post the whole thing later.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
Si no sabes adonde vas, es muy probable que acabes en otra parte.

  81. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-10T20:02:08Z

    On 2021-Sep-02, Alvaro Herrera wrote:

> On 2021-Sep-02, Rahila Syed wrote:
> 
> > After thinking about this, I think it is best to remove the entire table
> > from publication,
> > if a column specified in the column filter is dropped from the table.
> 
> Hmm, I think it would be cleanest to give responsibility to the user: if
> the column to be dropped is in the filter, then raise an error, aborting
> the drop.  Then it is up to them to figure out what to do.

I thought about this some more and realized that our earlier conclusions
were wrong or at least inconvenient.  I think that the best behavior if
you drop a column from a table is to remove the column from the
publication column list, and do nothing else.

Consider the case where you add a table to a publication without a
column filter, and later drop the column.  You don't get an error that
the relation is part of a publication; simply, the subscribers of that
publication will no longer receive that column.

Similarly for this case: if you add a table to a publication with a
column list, and later drop a column in that list, then you shouldn't
get an error either.  Simply the subscribers of that publication should
receive one column less.

Should be fairly quick to implement ... on it now.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"The problem with the facetime model is not just that it's demoralizing, but
that the people pretending to work interrupt the ones actually working."
                                                           (Paul Graham)

  82. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-10T21:08:24Z

    On 2021-Dec-10, Alvaro Herrera wrote:

> I thought about this some more and realized that our earlier conclusions
> were wrong or at least inconvenient.  I think that the best behavior if
> you drop a column from a table is to remove the column from the
> publication column list, and do nothing else.

> Should be fairly quick to implement ... on it now.

Actually it's not so easy to implement.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"No hay ausente sin culpa ni presente sin disculpa" (Prov. francés)

  83. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-13T17:44:11Z

    On 2021-Dec-10, Alvaro Herrera wrote:

> Actually it's not so easy to implement.

So I needed to add "sub object id" support for pg_publication_rel
objects in pg_depend / dependency.c.  What I have now is partial (the
describe routines need patched) but it's sufficient to show what's
needed.  In essence, we now set these depend entries with column
numbers, so that they can be dropped independently; when the drop comes,
the existing pg_publication_rel row is modified to cover the remaining
columns.  As far as I can tell, it works correctly.

There is one policy decision to make: what if ALTER TABLE drops the last
remaining column in the publication?  I opted to raise a specific error
in this case, though we could just the same opt to drop the relation
from the publication.  Are there opinions on this?

This version incorporates the fixups Peter submitted, plus some other
fixes of my own.  Notably, as Peter also mentioned, I changed
pg_publication_rel.prattrs to store int2vector rather than an array of
column names.  This makes for better behavior if columns are renamed and
things like that, and also we don't need to be so cautious about
quoting.  It does mean we need a slightly more complicated query in a
couple of spots, but that should be okay.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"Always assume the user will do much worse than the stupidest thing
you can imagine."                                (Julien PUYDT)

  84. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-13T17:47:50Z

    Hmm, I messed up the patch file I sent.  Here's the complete patch.


-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Doing what he did amounts to sticking his fingers under the hood of the
implementation; if he gets his fingers burnt, it's his problem."  (Tom Lane)

  85. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-13T21:19:54Z

    On 2021-Dec-13, Alvaro Herrera wrote:

> Hmm, I messed up the patch file I sent.  Here's the complete patch.

Actually, this requires even a bit more mess than this to be really
complete if we want to be strict about it.  The reason is that, with the
patch I just posted, we're creating a new type of representable object
that will need to have some way of making it through pg_identify_object,
pg_get_object_address, pg_identify_object_as_address.  This is only
visible as one tries to patch object_address.sql (auditability of DDL
operations being the goal).

I think this means we need a new OBJECT_PUBLICATION_REL_COLUMN value in
the ObjectType (paralelling OBJECT_COLUMN), and no new ObjectClass
value.  Looking now to confirm.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
"El que vive para el futuro es un iluso, y el que vive para el pasado,
un imbécil" (Luis Adler, "Los tripulantes de la noche")

  86. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-14T16:43:48Z

    On 2021-Dec-13, Alvaro Herrera wrote:

> I think this means we need a new OBJECT_PUBLICATION_REL_COLUMN value in
> the ObjectType (paralelling OBJECT_COLUMN), and no new ObjectClass
> value.  Looking now to confirm.

After working on this a little bit more, I realized that this is a bad
idea overall.  It causes lots of complications and it's just not worth
it.  So I'm back at my original thought that we need to throw an ERROR
at ALTER TABLE .. DROP COLUMN time if the column is part of a
replication column filter, and suggest the user to remove the column
from the filter first and reattempt the DROP COLUMN.

This means that we need to support changing the column list of a table
in a publication.  I'm looking at implementing some form of ALTER
PUBLICATION for that.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Find a bug in a program, and fix it, and the program will work today.
Show the program how to find and fix a bug, and the program
will work forever" (Oliver Silfridge)

  87. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2021-12-14T19:28:53Z

    On 12/14/21 17:43, Alvaro Herrera wrote:
> On 2021-Dec-13, Alvaro Herrera wrote:
> 
>> I think this means we need a new OBJECT_PUBLICATION_REL_COLUMN value in
>> the ObjectType (paralelling OBJECT_COLUMN), and no new ObjectClass
>> value.  Looking now to confirm.
> 
> After working on this a little bit more, I realized that this is a bad
> idea overall.  It causes lots of complications and it's just not worth
> it.  So I'm back at my original thought that we need to throw an ERROR
> at ALTER TABLE .. DROP COLUMN time if the column is part of a
> replication column filter, and suggest the user to remove the column
> from the filter first and reattempt the DROP COLUMN.
> 
> This means that we need to support changing the column list of a table
> in a publication.  I'm looking at implementing some form of ALTER
> PUBLICATION for that.
> 

Yeah. I think it's not clear if this should behave more like an index or 
a view. When an indexed column gets dropped we simply drop the index. 
But if you drop a column referenced by a view, we fail with an error. I 
think we should handle this more like a view, because publications are 
externally visible objects too (while indexes are pretty much just an 
implementation detail).

But why would it be easier not to add new object type? We still need to 
check there is no publication referencing the column - either you do 
that automatically through a dependency, or you do that by custom code. 
Using a dependency seems better to me, but I don't know what are the 
complications you mentioned.

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  88. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-14T19:35:23Z

    On 2021-Dec-14, Tomas Vondra wrote:

> Yeah. I think it's not clear if this should behave more like an index or a
> view. When an indexed column gets dropped we simply drop the index. But if
> you drop a column referenced by a view, we fail with an error. I think we
> should handle this more like a view, because publications are externally
> visible objects too (while indexes are pretty much just an implementation
> detail).

I agree -- I think it's more like a view than like an index.  (The
original proposal was that if you dropped a column that was part of the
column list of a relation in a publication, the entire relation is
dropped from the view, but that doesn't seem very friendly behavior --
you break the replication stream immediately if you do that, and the
only way to fix it is to send a fresh copy of the remaining subset of
columns.)

> But why would it be easier not to add new object type? We still need to
> check there is no publication referencing the column - either you do that
> automatically through a dependency, or you do that by custom code. Using a
> dependency seems better to me, but I don't know what are the complications
> you mentioned.

The problem is that we need a way to represent the object "column of a
table in a publication".  I found myself adding a lot of additional code
to support OBJECT_PUBLICATION_REL_COLUMN and that seemed like too much.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/

  89. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2021-12-14T21:13:49Z

    
On 12/14/21 20:35, Alvaro Herrera wrote:
> On 2021-Dec-14, Tomas Vondra wrote:
> 
>> Yeah. I think it's not clear if this should behave more like an index or a
>> view. When an indexed column gets dropped we simply drop the index. But if
>> you drop a column referenced by a view, we fail with an error. I think we
>> should handle this more like a view, because publications are externally
>> visible objects too (while indexes are pretty much just an implementation
>> detail).
> 
> I agree -- I think it's more like a view than like an index.  (The
> original proposal was that if you dropped a column that was part of the
> column list of a relation in a publication, the entire relation is
> dropped from the view, but that doesn't seem very friendly behavior --
> you break the replication stream immediately if you do that, and the
> only way to fix it is to send a fresh copy of the remaining subset of
> columns.)
> 

Right, that's my reasoning too.

>> But why would it be easier not to add new object type? We still need to
>> check there is no publication referencing the column - either you do that
>> automatically through a dependency, or you do that by custom code. Using a
>> dependency seems better to me, but I don't know what are the complications
>> you mentioned.
> 
> The problem is that we need a way to represent the object "column of a
> table in a publication".  I found myself adding a lot of additional code
> to support OBJECT_PUBLICATION_REL_COLUMN and that seemed like too much.
> 

My experience with dependencies is pretty limited, but can't we simply 
make a dependency between the whole publication and the column?

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  90. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2021-12-14T21:28:11Z

    Hi,

I went through the v9 patch, and I have a couple comments / questions. 
Apologies if some of this was already discussed earlier, it's hard to 
cross-check in such a long thread. Most of the comments are in 0002 to 
make it easier to locate, and it also makes proposed code changes 
clearer I think.

1) check_publication_add_relation - the "else" branch is not really 
needed, because the "if (replidentfull)" always errors-out

2) publication_add_relation has a FIXME about handling cases with 
different column list

So what's the right behavior for ADD TABLE with different column list? 
I'd say we should allow that, and that it should be mostly the same 
thing as adding/removing columns to the list incrementally, i.e. we 
should replace the column lists. We could also prohibit such changes, 
but that seems like a really annoying limitation, forcing people to 
remove/add the relation.

I added some comments to the attmap translation block, and replaced <0 
check with AttrNumberIsForUserDefinedAttr.

But I wonder if we could get rid of the offset, considering we're 
dealing with just user-defined attributes. That'd make the code clearer, 
but it would break if we're comparing it to other bitmaps with offsets. 
But I don't think we do.

3) I doubt "att_map" is the right name, though. AFAICS it's just a list 
of columns for the relation, not a map, right? So maybe attr_list?

4) AlterPublication talks about "publication status" for a column, but 
do we actually track that? Or what does that mean?

5) PublicationDropTables does a check

     if (pubrel->columns)
         ereport(ERROR,
                 errcode(ERRCODE_SYNTAX_ERROR),

Shouldn't this be prevented by the grammar, really? Also, it should be 
in regression tests.

6) Another thing that should be in the test is partitioned table with 
attribute mapping and column list, to see how map and attr_map interact.

7) There's a couple places doing this

     if (att_map != NULL &&
         !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
                        att_map) &&
         !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
                        idattrs) &&
         !replidentfull)

which is really hard to understand (even if we get rid of the offset), 
so maybe let's move that to a function with sensible name. Also, some 
places don't check indattrs - seems a bit suspicious.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  91. RE: Column Filtering in Logical Replication

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2021-12-16T09:28:28Z

    On Tues, Dec 14, 2021 1:48 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> Hmm, I messed up the patch file I sent.  Here's the complete patch.
> 

Hi,

I have a minor question about the replica identity check of this patch.

+check_publication_add_relation(Relation targetrel, Bitmapset *columns)
...
+			idattrs = RelationGetIndexAttrBitmap(targetrel,
+												 INDEX_ATTR_BITMAP_IDENTITY_KEY);
+			if (!bms_is_subset(idattrs, columns))
+				ereport(ERROR,
+						errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+						errmsg("invalid column list for publishing relation \"%s\"",
+							   RelationGetRelationName(targetrel)),
+						errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+

The patch ensures all columns of RT are in column list when CREATE/ALTER
publication, but it seems doesn't prevent user from changing the replica
identity or dropping the index used in replica identity. Do we also need to
check those cases ?

Best regards,
Hou zj

  92. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-16T17:54:59Z

    On 2021-Dec-16, houzj.fnst@fujitsu.com wrote:

> The patch ensures all columns of RT are in column list when CREATE/ALTER
> publication, but it seems doesn't prevent user from changing the replica
> identity or dropping the index used in replica identity. Do we also need to
> check those cases ?

Yes, we do.  As it happens, I spent a couple of hours yesterday writing
code for that, at least partially.  I haven't yet checked what happens
with cases like REPLICA NOTHING, or REPLICA INDEX <xyz> and then
dropping that index.

My initial ideas were a bit wrong BTW: I thought we should check the
combination of column lists in all publications (a bitwise-OR of column
bitmaps, so to speak).  But conceptually that's wrong: we need to check
the column list of each publication individually instead.  Otherwise, if
you wanted to hide a column from some publication but that column was
part of the replica identity, there'd be no way to identify the tuple in
the replica.  (Or, if the pgouput code disobeys the column list and
sends the replica identity even if it's not in the column list, then
you'd be potentially publishing data that you wanted to hide.)

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/

  93. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-16T20:10:58Z

    On 2021-Dec-14, Tomas Vondra wrote:

> 7) There's a couple places doing this
> 
>     if (att_map != NULL &&
>         !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
>                        att_map) &&
>         !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
>                        idattrs) &&
>         !replidentfull)
> 
> which is really hard to understand (even if we get rid of the offset), so
> maybe let's move that to a function with sensible name. Also, some places
> don't check indattrs - seems a bit suspicious.

It is indeed pretty hard to read ... but I think this is completely
unnecessary.  Any column that is part of the identity should have been
included in the column filter, so there is no need to check for the
identity attributes separately.  Testing just for the columns in the
filter ought to be sufficient; and the cases "if att_map NULL" and "is
replica identity FULL" are also equivalent, because in the case of FULL,
you're disallowed from setting a column list.  So this whole thing can
be reduced to just this:

if (att_map != NULL && !bms_is_member(att->attnum, att_map))
       continue;	/* that is, don't send this attribute */

so I don't think this merits a separate function.

[ says he, after already trying to write said function ]

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Before you were born your parents weren't as boring as they are now. They
got that way paying your bills, cleaning up your room and listening to you
tell them how idealistic you are."  -- Charles J. Sykes' advice to teenagers

  94. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-12-17T04:47:33Z

    On Wed, Dec 15, 2021 at 1:05 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Dec-14, Tomas Vondra wrote:
>
> > Yeah. I think it's not clear if this should behave more like an index or a
> > view. When an indexed column gets dropped we simply drop the index. But if
> > you drop a column referenced by a view, we fail with an error. I think we
> > should handle this more like a view, because publications are externally
> > visible objects too (while indexes are pretty much just an implementation
> > detail).
>
> I agree -- I think it's more like a view than like an index.  (The
> original proposal was that if you dropped a column that was part of the
> column list of a relation in a publication, the entire relation is
> dropped from the view,
>

I think in the above sentence, you mean to say "dropped from the
publication". So, IIUC, you are proposing that if one drops a column
that was part of the column list of a relation in a publication, an
error will be raised. Also, if the user specifies CASCADE in Alter
Table ... Drop Column, then we drop the relation from publication. Is
that right? BTW, this is somewhat on the lines of what row_filter
patch is also doing where if the user drops the column that was part
of row_filter for a relation in publication, we give an error and if
the user tries to drop the column with CASCADE then the relation is
removed from the publication.

-- 
With Regards,
Amit Kapila.

  95. Re: Column Filtering in Logical Replication

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2021-12-17T07:03:33Z

    On 17.12.21 05:47, Amit Kapila wrote:
> I think in the above sentence, you mean to say "dropped from the
> publication". So, IIUC, you are proposing that if one drops a column
> that was part of the column list of a relation in a publication, an
> error will be raised. Also, if the user specifies CASCADE in Alter
> Table ... Drop Column, then we drop the relation from publication. Is
> that right? BTW, this is somewhat on the lines of what row_filter
> patch is also doing where if the user drops the column that was part
> of row_filter for a relation in publication, we give an error and if
> the user tries to drop the column with CASCADE then the relation is
> removed from the publication.

That looks correct.  Consider how triggers behave: Dropping a column 
that a trigger uses (either in UPDATE OF or a WHEN condition) errors 
with RESTRICT and drops the trigger with CASCADE.

  96. RE: Column Filtering in Logical Replication

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2021-12-17T09:46:14Z

    On Friday, December 17, 2021 1:55 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> On 2021-Dec-16, houzj.fnst@fujitsu.com wrote:
> 
> > The patch ensures all columns of RT are in column list when
> > CREATE/ALTER publication, but it seems doesn't prevent user from
> > changing the replica identity or dropping the index used in replica
> > identity. Do we also need to check those cases ?
> 
> Yes, we do.  As it happens, I spent a couple of hours yesterday writing code for
> that, at least partially.  I haven't yet checked what happens with cases like
> REPLICA NOTHING, or REPLICA INDEX <xyz> and then dropping that index.
> 
> My initial ideas were a bit wrong BTW: I thought we should check the
> combination of column lists in all publications (a bitwise-OR of column bitmaps,
> so to speak).  But conceptually that's wrong: we need to check the column list
> of each publication individually instead.  Otherwise, if you wanted to hide a
> column from some publication but that column was part of the replica identity,
> there'd be no way to identify the tuple in the replica.  (Or, if the pgouput code
> disobeys the column list and sends the replica identity even if it's not in the
> column list, then you'd be potentially publishing data that you wanted to hide.)

Thanks for the explanation.

Apart from ALTER REPLICA IDENTITY and DROP INDEX, I think there could be
some other cases we need to handle for the replica identity check:

1)
When adding a partitioned table with column list to the publication, I think we
need to check the RI of all its leaf partition. Because the RI on the partition
is the one actually takes effect.

2)
ALTER TABLE ADD PRIMARY KEY;
ALTER TABLE DROP CONSTRAINT "PRIMAEY KEY";

If the replica identity is default, it will use the primary key. we might also
need to prevent user from adding or removing primary key in this case.


Based on the above cases, the RI check seems could bring considerable amount of
code. So, how about we follow what we already did in CheckCmdReplicaIdentity(),
we can put the check for RI in that function, so that we can cover all the
cases and reduce the code change. And if we are worried about the cost of do
the check for UPDATE and DELETE every time, we can also save the result in the
relcache. It's safe because every operation change the RI will invalidate the
relcache. We are using this approach in row filter patch to make sure all
columns in row filter expression are part of RI.

Best regards,
Hou zj

  97. Re: Column Filtering in Logical Replication

    Rahila Syed <rahilasyed90@gmail.com> — 2021-12-17T12:09:37Z

    Hi,

Thank you for updating the patch. The regression tests and tap tests pass
with v9 patch.


>
> After working on this a little bit more, I realized that this is a bad
> idea overall.  It causes lots of complications and it's just not worth
> it.  So I'm back at my original thought that we need to throw an ERROR
> at ALTER TABLE .. DROP COLUMN time if the column is part of a
> replication column filter, and suggest the user to remove the column
> from the filter first and reattempt the DROP COLUMN.
>
> This means that we need to support changing the column list of a table
> in a publication.  I'm looking at implementing some form of ALTER
> PUBLICATION for that.
>
>
I think right now the patch contains support only for ALTER PUBLICATION..
ADD TABLE with column filters.
In order to achieve changing the column lists of a published table, I think
we can extend the
ALTER TABLE ..SET TABLE syntax to support specification of column list.

So this whole thing can
> be reduced to just this:


if (att_map != NULL && !bms_is_member(att->attnum, att_map))
>        continue;        /* that is, don't send this attribute */


I agree the condition can be shortened now. The long if condition was
included because initially the feature
allowed specifying filters without replica identity columns(sent those
columns internally without user
having to specify).

 900 +        * the table is partitioned.  Run a recursive query to iterate
> through all
>  901 +        * the parents of the partition and retreive the record for
> the parent
>  902 +        * that exists in pg_publication_rel.
>  903 +        */


The above comment in fetch_remote_table_info() can be changed as the
recursive query
is no longer used.

Thank you,
Rahila Syed

  98. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-17T15:58:50Z

    On 2021-Dec-17, Rahila Syed wrote:

> > This means that we need to support changing the column list of a
> > table in a publication.  I'm looking at implementing some form of
> > ALTER PUBLICATION for that.
>
> I think right now the patch contains support only for ALTER
> PUBLICATION..  ADD TABLE with column filters.  In order to achieve
> changing the column lists of a published table, I think we can extend
> the ALTER TABLE ..SET TABLE syntax to support specification of column
> list.

Yeah, that's what I was thinking too.

> > So this whole thing can be reduced to just this:
> 
> > if (att_map != NULL && !bms_is_member(att->attnum, att_map))
> >        continue;        /* that is, don't send this attribute */
> 
> I agree the condition can be shortened now. The long if condition was
> included because initially the feature allowed specifying filters
> without replica identity columns(sent those columns internally without
> user having to specify).

Ah, true, I had forgotten that.  Thanks.

> >  900 +        * the table is partitioned.  Run a recursive query to iterate through all
> >  901 +        * the parents of the partition and retreive the record for the parent
> >  902 +        * that exists in pg_publication_rel.
> >  903 +        */
> 
> The above comment in fetch_remote_table_info() can be changed as the
> recursive query is no longer used.

Oh, of course.

I'll finish some loose ends and submit a v10, but it's still not final.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"Right now the sectors on the hard disk run clockwise, but I heard a rumor that
you can squeeze 0.2% more throughput by running them counterclockwise.
It's worth the effort. Recommended."  (Gerry Pourwelle)

  99. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-17T21:07:18Z

    So I've been thinking about this as a "security" item (you can see my
comments to that effect sprinkled all over this thread), in the sense
that if a publication "hides" some column, then the replica just won't
get access to it.  But in reality that's mistaken: the filtering that
this patch implements is done based on the queries that *the replica*
executes at its own volition; if the replica decides to ignore the list
of columns, it'll be able to get all columns.  All it takes is an
uncooperative replica in order for the lot of data to be exposed anyway.

If the server has a *separate* security mechanism to hide the columns
(per-column privs), it is that feature that will protect the data, not
the logical-replication-feature to filter out columns.


This led me to realize that the replica-side code in tablesync.c is
totally oblivious to what's the publication through which a table is
being received from in the replica.  So we're not aware of a replica
being exposed only a subset of columns through some specific
publication; and a lot more hacking is needed than this patch does, in
order to be aware of which publications are being used.

I'm going to have a deeper look at this whole thing.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/

  100. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2021-12-17T21:57:56Z

    
On 12/17/21 22:07, Alvaro Herrera wrote:
> So I've been thinking about this as a "security" item (you can see my
> comments to that effect sprinkled all over this thread), in the sense
> that if a publication "hides" some column, then the replica just won't
> get access to it.  But in reality that's mistaken: the filtering that
> this patch implements is done based on the queries that *the replica*
> executes at its own volition; if the replica decides to ignore the list
> of columns, it'll be able to get all columns.  All it takes is an
> uncooperative replica in order for the lot of data to be exposed anyway.
> 

Interesting, I haven't really looked at this as a security feature. And 
in my experience if something is not carefully designed to be secure 
from the get go, it's really hard to add that bit later ...

You say it's the replica making the decisions, but my mental model is 
it's the publisher decoding the data for a given list of publications 
(which indeed is specified by the subscriber). But the subscriber can't 
tweak the definition of publications, right? Or what do you mean by 
queries executed by the replica? What are the gap?

> If the server has a *separate* security mechanism to hide the columns
> (per-column privs), it is that feature that will protect the data, not
> the logical-replication-feature to filter out columns.
> 

Right. Although I haven't thought about how logical decoding interacts 
with column privileges. I don't think logical decoding actually checks 
column privileges - I certainly don't recall any ACL checks in 
src/backend/replication ...

AFAIK we only really check privileges during initial sync (when creating 
the slot and copying data), but then we keep replicating data even if 
the privilege gets revoked for the table/column. In principle the 
replication role is pretty close to superuser.

> 
> This led me to realize that the replica-side code in tablesync.c is
> totally oblivious to what's the publication through which a table is
> being received from in the replica.  So we're not aware of a replica
> being exposed only a subset of columns through some specific
> publication; and a lot more hacking is needed than this patch does, in
> order to be aware of which publications are being used.
> 
> I'm going to have a deeper look at this whole thing.
> 

Does that mean we currently sync all the columns in the initial sync, 
and only start filtering columns later while decoding transactions?


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  101. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-18T01:34:16Z

    On 2021-Dec-17, Tomas Vondra wrote:

> On 12/17/21 22:07, Alvaro Herrera wrote:
> > So I've been thinking about this as a "security" item (you can see my
> > comments to that effect sprinkled all over this thread), in the sense
> > that if a publication "hides" some column, then the replica just won't
> > get access to it.  But in reality that's mistaken: the filtering that
> > this patch implements is done based on the queries that *the replica*
> > executes at its own volition; if the replica decides to ignore the list
> > of columns, it'll be able to get all columns.  All it takes is an
> > uncooperative replica in order for the lot of data to be exposed anyway.
> 
> Interesting, I haven't really looked at this as a security feature. And in
> my experience if something is not carefully designed to be secure from the
> get go, it's really hard to add that bit later ...

I guess the way to really harden replication is to use the GRANT system
at the publisher's side to restrict access for the replication user.
This would provide actual security.  So you're right that I seem to be
barking at the wrong tree ...  maybe I need to give a careful look at
the documentation for logical replication to understand what is being
offered, and to make sure that we explicitly indicate that limiting the
column list does not provide any actual security.

> You say it's the replica making the decisions, but my mental model is it's
> the publisher decoding the data for a given list of publications (which
> indeed is specified by the subscriber). But the subscriber can't tweak the
> definition of publications, right? Or what do you mean by queries executed
> by the replica? What are the gap?

I am thinking in somebody modifying the code that the replica runs, so
that it ignores the column list that the publication has been configured
to provide; instead of querying only those columns, it would query all
columns.

> > If the server has a *separate* security mechanism to hide the columns
> > (per-column privs), it is that feature that will protect the data, not
> > the logical-replication-feature to filter out columns.
> 
> Right. Although I haven't thought about how logical decoding interacts with
> column privileges. I don't think logical decoding actually checks column
> privileges - I certainly don't recall any ACL checks in
> src/backend/replication ...

Well, in practice if you're confronted with a replica that's controlled
by a malicious user that can tweak its behavior, then replica-side
privilege checking won't do anything useful.

> > This led me to realize that the replica-side code in tablesync.c is
> > totally oblivious to what's the publication through which a table is
> > being received from in the replica.  So we're not aware of a replica
> > being exposed only a subset of columns through some specific
> > publication; and a lot more hacking is needed than this patch does, in
> > order to be aware of which publications are being used.

> Does that mean we currently sync all the columns in the initial sync, and
> only start filtering columns later while decoding transactions?

No, it does filter the list of columns in the initial sync.  But the
current implementation is bogus, because it obtains the list of *all*
publications in which the table is published, not just the ones that the
subscription is configured to get data from.  And the sync code doesn't
receive the list of publications.  We need more thorough patching of the
sync code to close that hole.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/

  102. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2021-12-18T01:59:00Z

    
On 12/18/21 02:34, Alvaro Herrera wrote:
> On 2021-Dec-17, Tomas Vondra wrote:
> 
>> On 12/17/21 22:07, Alvaro Herrera wrote:
>>> So I've been thinking about this as a "security" item (you can see my
>>> comments to that effect sprinkled all over this thread), in the sense
>>> that if a publication "hides" some column, then the replica just won't
>>> get access to it.  But in reality that's mistaken: the filtering that
>>> this patch implements is done based on the queries that *the replica*
>>> executes at its own volition; if the replica decides to ignore the list
>>> of columns, it'll be able to get all columns.  All it takes is an
>>> uncooperative replica in order for the lot of data to be exposed anyway.
>>
>> Interesting, I haven't really looked at this as a security feature. And in
>> my experience if something is not carefully designed to be secure from the
>> get go, it's really hard to add that bit later ...
> 
> I guess the way to really harden replication is to use the GRANT system
> at the publisher's side to restrict access for the replication user.
> This would provide actual security.  So you're right that I seem to be
> barking at the wrong tree ...  maybe I need to give a careful look at
> the documentation for logical replication to understand what is being
> offered, and to make sure that we explicitly indicate that limiting the
> column list does not provide any actual security.
> 
>> You say it's the replica making the decisions, but my mental model is it's
>> the publisher decoding the data for a given list of publications (which
>> indeed is specified by the subscriber). But the subscriber can't tweak the
>> definition of publications, right? Or what do you mean by queries executed
>> by the replica? What are the gap?
> 
> I am thinking in somebody modifying the code that the replica runs, so
> that it ignores the column list that the publication has been configured
> to provide; instead of querying only those columns, it would query all
> columns.
> 
>>> If the server has a *separate* security mechanism to hide the columns
>>> (per-column privs), it is that feature that will protect the data, not
>>> the logical-replication-feature to filter out columns.
>>
>> Right. Although I haven't thought about how logical decoding interacts with
>> column privileges. I don't think logical decoding actually checks column
>> privileges - I certainly don't recall any ACL checks in
>> src/backend/replication ...
> 
> Well, in practice if you're confronted with a replica that's controlled
> by a malicious user that can tweak its behavior, then replica-side
> privilege checking won't do anything useful.
> 

I don't follow. Surely the decoding happens on the primary node, right? 
Which is where the ACL checks would happen, using the role the 
replication connection is opened with.

>>> This led me to realize that the replica-side code in tablesync.c is
>>> totally oblivious to what's the publication through which a table is
>>> being received from in the replica.  So we're not aware of a replica
>>> being exposed only a subset of columns through some specific
>>> publication; and a lot more hacking is needed than this patch does, in
>>> order to be aware of which publications are being used.
> 
>> Does that mean we currently sync all the columns in the initial sync, and
>> only start filtering columns later while decoding transactions?
> 
> No, it does filter the list of columns in the initial sync.  But the
> current implementation is bogus, because it obtains the list of *all*
> publications in which the table is published, not just the ones that the
> subscription is configured to get data from.  And the sync code doesn't
> receive the list of publications.  We need more thorough patching of the
> sync code to close that hole.

Ah, got it. Thanks for the explanation. Yeah, that makes no sense.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  103. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-12-18T03:52:05Z

    On Sat, Dec 18, 2021 at 7:04 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Dec-17, Tomas Vondra wrote:
>
> > On 12/17/21 22:07, Alvaro Herrera wrote:
> > > So I've been thinking about this as a "security" item (you can see my
> > > comments to that effect sprinkled all over this thread), in the sense
> > > that if a publication "hides" some column, then the replica just won't
> > > get access to it.  But in reality that's mistaken: the filtering that
> > > this patch implements is done based on the queries that *the replica*
> > > executes at its own volition; if the replica decides to ignore the list
> > > of columns, it'll be able to get all columns.  All it takes is an
> > > uncooperative replica in order for the lot of data to be exposed anyway.
> >
> > Interesting, I haven't really looked at this as a security feature. And in
> > my experience if something is not carefully designed to be secure from the
> > get go, it's really hard to add that bit later ...
>
> I guess the way to really harden replication is to use the GRANT system
> at the publisher's side to restrict access for the replication user.
> This would provide actual security.  So you're right that I seem to be
> barking at the wrong tree ...  maybe I need to give a careful look at
> the documentation for logical replication to understand what is being
> offered, and to make sure that we explicitly indicate that limiting the
> column list does not provide any actual security.
>

IIRC, the use cases as mentioned by other databases (like Oracle) are
(a) this helps when the target table doesn't have the same set of
columns or (b) when the columns contain some sensitive information
like personal identification number, etc. I think there could be a
side benefit in this which comes from the fact that the lesser data
will flow across the network which could lead to faster replication
especially when the user filters large column data.

-- 
With Regards,
Amit Kapila.

  104. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2021-12-18T04:07:02Z

    On Fri, Dec 17, 2021 at 3:16 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
>
> On Friday, December 17, 2021 1:55 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> > On 2021-Dec-16, houzj.fnst@fujitsu.com wrote:
> >
> > > The patch ensures all columns of RT are in column list when
> > > CREATE/ALTER publication, but it seems doesn't prevent user from
> > > changing the replica identity or dropping the index used in replica
> > > identity. Do we also need to check those cases ?
> >
> > Yes, we do.  As it happens, I spent a couple of hours yesterday writing code for
> > that, at least partially.  I haven't yet checked what happens with cases like
> > REPLICA NOTHING, or REPLICA INDEX <xyz> and then dropping that index.
> >
> > My initial ideas were a bit wrong BTW: I thought we should check the
> > combination of column lists in all publications (a bitwise-OR of column bitmaps,
> > so to speak).  But conceptually that's wrong: we need to check the column list
> > of each publication individually instead.  Otherwise, if you wanted to hide a
> > column from some publication but that column was part of the replica identity,
> > there'd be no way to identify the tuple in the replica.  (Or, if the pgouput code
> > disobeys the column list and sends the replica identity even if it's not in the
> > column list, then you'd be potentially publishing data that you wanted to hide.)
>
> Thanks for the explanation.
>
> Apart from ALTER REPLICA IDENTITY and DROP INDEX, I think there could be
> some other cases we need to handle for the replica identity check:
>
> 1)
> When adding a partitioned table with column list to the publication, I think we
> need to check the RI of all its leaf partition. Because the RI on the partition
> is the one actually takes effect.
>
> 2)
> ALTER TABLE ADD PRIMARY KEY;
> ALTER TABLE DROP CONSTRAINT "PRIMAEY KEY";
>
> If the replica identity is default, it will use the primary key. we might also
> need to prevent user from adding or removing primary key in this case.
>
>
> Based on the above cases, the RI check seems could bring considerable amount of
> code. So, how about we follow what we already did in CheckCmdReplicaIdentity(),
> we can put the check for RI in that function, so that we can cover all the
> cases and reduce the code change. And if we are worried about the cost of do
> the check for UPDATE and DELETE every time, we can also save the result in the
> relcache. It's safe because every operation change the RI will invalidate the
> relcache. We are using this approach in row filter patch to make sure all
> columns in row filter expression are part of RI.
>

Another point related to RI is that this patch seems to restrict
specifying the RI columns in the column filter list irrespective of
publish action. Do we need to have such a restriction if the
publication publishes 'insert' or 'truncate'?

-- 
With Regards,
Amit Kapila.

  105. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-18T16:51:56Z

    On 2021-Dec-18, Tomas Vondra wrote:

> On 12/18/21 02:34, Alvaro Herrera wrote:
> > On 2021-Dec-17, Tomas Vondra wrote:

> > > > If the server has a *separate* security mechanism to hide the
> > > > columns (per-column privs), it is that feature that will protect
> > > > the data, not the logical-replication-feature to filter out
> > > > columns.
> > > 
> > > Right. Although I haven't thought about how logical decoding
> > > interacts with column privileges. I don't think logical decoding
> > > actually checks column privileges - I certainly don't recall any
> > > ACL checks in src/backend/replication ...
> > 
> > Well, in practice if you're confronted with a replica that's
> > controlled by a malicious user that can tweak its behavior, then
> > replica-side privilege checking won't do anything useful.
> 
> I don't follow. Surely the decoding happens on the primary node,
> right?  Which is where the ACL checks would happen, using the role the
> replication connection is opened with.

I think you do follow.  Yes, the decoding happens on the primary node,
and the security checks should occur in the primary node, because to do
otherwise is folly(*).  Which means that column filtering, being a
replica-side feature, is *not* a security feature.  I was mistaken about
it, is all.  If you want security, you need to use column-level
privileges, as you say.

(*) The checks *must* occur in the primary side, because the primary
does not control the code that runs in the replica side.  The primary
must treat the replica as running potentially hostile code.  Trying to
defend against that is not practical.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/

  106. Re: Column Filtering in Logical Replication

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2021-12-20T13:53:46Z

    On 17.12.21 22:07, Alvaro Herrera wrote:
> So I've been thinking about this as a "security" item (you can see my
> comments to that effect sprinkled all over this thread), in the sense
> that if a publication "hides" some column, then the replica just won't
> get access to it.  But in reality that's mistaken: the filtering that
> this patch implements is done based on the queries that *the replica*
> executes at its own volition; if the replica decides to ignore the list
> of columns, it'll be able to get all columns.  All it takes is an
> uncooperative replica in order for the lot of data to be exposed anyway.

During normal replication, the publisher should only send the columns 
that are configured to be part of the publication.  So I don't see a 
problem there.

During the initial table sync, the subscriber indeed can construct any 
COPY command.  We could maybe replace this with a more customized COPY 
command variant, like COPY table OF publication TO STDOUT.

But right now the subscriber is sort of assumed to have access to 
everything on the publisher anyway, so I doubt that this is the only 
problem.  But it's worth considering.

  107. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-27T17:06:18Z

    Determining that an array has a NULL element seems convoluted.  I ended
up with this query, where comparing the result of array_positions() with
an empty array does that.  If anybody knows of a simpler way, or any
situations in which this fails, I'm all ears.

with published_cols as (
        select case when
                pg_catalog.array_positions(pg_catalog.array_agg(unnest), null) <> '{}' then null else
                pg_catalog.array_agg(distinct unnest order by unnest) end AS attrs
        from pg_catalog.pg_publication p join
                pg_catalog.pg_publication_rel pr on (p.oid = pr.prpubid) left join
                unnest(prattrs) on (true)
        where prrelid = 38168 and p.pubname in ('pub1', 'pub2')
)
SELECT a.attname,
       a.atttypid,
       a.attnum = ANY(i.indkey)
  FROM pg_catalog.pg_attribute a
  LEFT JOIN pg_catalog.pg_index i
       ON (i.indexrelid = pg_get_replica_identity_index(38168)),
     published_cols
 WHERE a.attnum > 0::pg_catalog.int2
   AND NOT a.attisdropped and a.attgenerated = ''
   AND a.attrelid = 38168
   AND (published_cols.attrs IS NULL OR attnum = ANY(published_cols.attrs))
 ORDER BY a.attnum;

This returns all columns if at least one publication has a NULL prattrs,
or only the union of columns listed in all publications, if all
publications have a list of columns.

(I was worried about obtaining the list of publications, but it turns
out that it's already as a convenient list of OIDs in the MySubscription
struct.)

With this, we can remove the second query added by Rahila's original patch to
filter out nonpublished columns.

I still need to add pg_partition_tree() in order to search for
publications containing a partition ancestor.  I'm not yet sure what
happens (and what *should* happen) if an ancestor is part of a
publication and the partition is also part of a publication, and the
column lists differ.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
Al principio era UNIX, y UNIX habló y dijo: "Hello world\n".
No dijo "Hello New Jersey\n", ni "Hello USA\n".

  108. Re: Column Filtering in Logical Replication

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-12-27T17:38:29Z

    Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> Determining that an array has a NULL element seems convoluted.  I ended
> up with this query, where comparing the result of array_positions() with
> an empty array does that.  If anybody knows of a simpler way, or any
> situations in which this fails, I'm all ears.

Maybe better to rethink why we allow elements of prattrs to be null?

			regards, tom lane

  109. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-27T18:31:46Z

    On 2021-Dec-27, Tom Lane wrote:

> Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> > Determining that an array has a NULL element seems convoluted.  I ended
> > up with this query, where comparing the result of array_positions() with
> > an empty array does that.  If anybody knows of a simpler way, or any
> > situations in which this fails, I'm all ears.
> 
> Maybe better to rethink why we allow elements of prattrs to be null?

What I'm doing is an unnest of all arrays and then aggregating them
back into a single array.  If one array is null, the resulting aggregate
contains a null element.

Hmm, maybe I can in parallel do a bool_or() aggregate of "array is null" to
avoid that.  ... ah yes, that works:

with published_cols as (
        select pg_catalog.bool_or(pr.prattrs is null) as all_columns,
                pg_catalog.array_agg(distinct unnest order by unnest) AS attrs
        from pg_catalog.pg_publication p join
                pg_catalog.pg_publication_rel pr on (p.oid = pr.prpubid) left join
                unnest(prattrs) on (true)
        where prrelid = :table and p.pubname in ('pub1', 'pub2')
)
SELECT a.attname,
       a.atttypid,
       a.attnum = ANY(i.indkey)
  FROM pg_catalog.pg_attribute a
  LEFT JOIN pg_catalog.pg_index i
       ON (i.indexrelid = pg_get_replica_identity_index(:table)),
     published_cols
 WHERE a.attnum > 0::pg_catalog.int2
   AND NOT a.attisdropped and a.attgenerated = ''
   AND a.attrelid = :table
   AND (all_columns OR attnum = ANY(published_cols.attrs))
 ORDER BY a.attnum ;

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/

  110. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-28T21:04:27Z

    OK, getting closer now.  I've fixed the code to filter them column list
during the initial sync, and added some more tests for code that wasn't
covered.

There are still some XXX comments.  The one that bothers me most is the
lack of an implementation that allows changing the column list in a
publication without having to remove the table from the publication
first.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"I'm always right, but sometimes I'm more right than other times."
                                                  (Linus Torvalds)

  111. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-30T00:15:08Z

    On 2021-Dec-28, Alvaro Herrera wrote:

> There are still some XXX comments.  The one that bothers me most is the
> lack of an implementation that allows changing the column list in a
> publication without having to remove the table from the publication
> first.

OK, I made some progress on this front; I added new forms of ALTER
PUBLICATION to support it:

ALTER PUBLICATION pub1 ALTER TABLE tbl SET COLUMNS (a, b, c);
ALTER PUBLICATION pub1 ALTER TABLE tbl SET COLUMNS ALL;

(not wedded to this syntax; other suggestions welcome)

In order to implement it I changed the haphazardly chosen use of
DEFELEM actions to a new enum.  I also noticed that the division of
labor between pg_publication.c and publicationcmds.c is quite broken
(code to translate column names to numbers is in the former, should be
in the latter; some code that deals with pg_publication tuples is in the
latter, should be in the former, such as CreatePublication,
AlterPublicationOptions).

This new stuff is not yet finished.  For example I didn't refactor
handling of REPLICA IDENTITY, so the new command does not correctly
check everything, such as the REPLICA IDENTITY FULL stuff.  Also, no
tests have been added yet.  In manual tests it seems to behave as
expected.

I noticed that prattrs is inserted in user-specified order instead of
catalog order, which is innocuous but quite weird.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"No renuncies a nada. No te aferres a nada."

  112. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-30T20:21:28Z

    On 2021-Dec-29, Alvaro Herrera wrote:

> This new stuff is not yet finished.  For example I didn't refactor
> handling of REPLICA IDENTITY, so the new command does not correctly
> check everything, such as the REPLICA IDENTITY FULL stuff.  Also, no
> tests have been added yet.  In manual tests it seems to behave as
> expected.

Fixing the lack of check for replica identity full didn't really require
much refactoring, so I did it that way.

I split it with some trivial fixes that can be committed separately
ahead of time.  I'm thinking in committing 0001 later today, perhaps
0002 tomorrow.  The interesting part is 0003.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/

  113. Re: Column Filtering in Logical Replication

    Justin Pryzby <pryzby@telsasoft.com> — 2021-12-30T22:16:34Z

    > +	bool		am_partition = false;
>...
>	Assert(!isnull);
>	lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
>	Assert(!isnull);
> +	am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));

I think this needs to be GetBool.
You should Assert(!isnull) like the others.
Also, I think it doesn't need to be initialized to "false".

> +		/*
> +		 * Even if the user listed all columns in the column list, we cannot
> +		 * allow a column list to be specified when REPLICA IDENTITY is FULL;
> +		 * that would cause problems if a new column is added later, because
> +		 * that could would have to be included (because of being part of the

could would is wrong

> +	/*
> +	 * Translate list of columns to attnums. We prohibit system attributes and
> +	 * make sure there are no duplicate columns.
> +	 *
> +	 */

extraneous line

> +/*
> + * Gets a list of OIDs of all column-partial publications of the given
> + * relation, that is, those that specify a column list.

I would call this a "partial-column" publication.

> +					errmsg("cannot set REPLICA IDENTITY FULL when column-partial publications exist"));
> +	 * Check column-partial publications.  All publications have to include all

same

> +	/*
> +	 * Store the column names only if they are contained in column filter

period(.)

> +	 * LogicalRepRelation will only contain attributes corresponding to those
> +	 * specficied in column filters.

specified

> --- a/src/include/catalog/pg_publication_rel.h
> +++ b/src/include/catalog/pg_publication_rel.h
> @@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
>  	Oid			oid;			/* oid */
>  	Oid			prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
>  	Oid			prrelid BKI_LOOKUP(pg_class);	/* Oid of the relation */
> +#ifdef CATALOG_VARLEN
> +	int2vector	prattrs;		/* Variable length field starts here */
> +#endif

The language in the pre-existing comments is better:
	/* variable-length fields start here */

> @@ -791,12 +875,13 @@ fetch_remote_table_info(char *nspname, char *relname,
>
>                 ExecClearTuple(slot);
>         }
> +
>         ExecDropSingleTupleTableSlot(slot);
> +       walrcv_clear_result(res);
> +       pfree(cmd.data);
>
>         lrel->natts = natt;
>
> -       walrcv_clear_result(res);
> -       pfree(cmd.data);
>  }

The blank line after "lrel->natts = natt;" should be removed.

  114. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-12-30T23:32:29Z

    On 2021-Dec-30, Justin Pryzby wrote:

Thank you!  I've incorporated your proposed fixes.

> > +		/*
> > +		 * Even if the user listed all columns in the column list, we cannot
> > +		 * allow a column list to be specified when REPLICA IDENTITY is FULL;
> > +		 * that would cause problems if a new column is added later, because
> > +		 * that could would have to be included (because of being part of the
> 
> could would is wrong

Hah, yeah, this was "that column would".

> > + * Gets a list of OIDs of all column-partial publications of the given
> > + * relation, that is, those that specify a column list.
> 
> I would call this a "partial-column" publication.

OK, done that way.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/

  115. Re: Column Filtering in Logical Replication

    Justin Pryzby <pryzby@telsasoft.com> — 2021-12-31T16:32:27Z

    > @@ -5963,8 +5967,20 @@ describePublications(const char *pattern)
>  		{
>  			/* Get the tables for the specified publication */
>  			printfPQExpBuffer(&buf,
> -							  "SELECT n.nspname, c.relname\n"
> -							  "FROM pg_catalog.pg_class c,\n"
> +							  "SELECT n.nspname, c.relname, \n");
> +			if (pset.sversion >= 150000)
> +				appendPQExpBufferStr(&buf,
> +									 "       CASE WHEN pr.prattrs IS NOT NULL THEN\n"
> +									 "       pg_catalog.array_to_string"
> +									 "(ARRAY(SELECT attname\n"
> +									 "         FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::int[], 1)) s,\n"
> +									 "              pg_catalog.pg_attribute\n"
> +									 "        WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
> +									 "       ELSE NULL END AS columns");
> +			else
> +				appendPQExpBufferStr(&buf, "NULL as columns");
> +			appendPQExpBuffer(&buf,
> +							  "\nFROM pg_catalog.pg_class c,\n"
>  							  "     pg_catalog.pg_namespace n,\n"
>  							  "     pg_catalog.pg_publication_rel pr\n"
>  							  "WHERE c.relnamespace = n.oid\n"

I suppose this should use pr.prattrs::pg_catalog.int2[] ?

Did the DatumGetBool issue expose a deficiency in testing ?
I think the !am_partition path was never being hit.

-- 
Justin

  116. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-01-03T14:31:39Z

    On 2021-Dec-31, Justin Pryzby wrote:

> > @@ -5963,8 +5967,20 @@ describePublications(const char *pattern)
> >  		{
> > +									 "       CASE WHEN pr.prattrs IS NOT NULL THEN\n"
> > +									 "       pg_catalog.array_to_string"
> > +									 "(ARRAY(SELECT attname\n"
> > +									 "         FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::int[], 1)) s,\n"
> > +									 "              pg_catalog.pg_attribute\n"
> > +									 "        WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
> > +									 "       ELSE NULL END AS columns");

> I suppose this should use pr.prattrs::pg_catalog.int2[] ?

True.  Changed that.

Another change in this v15 is that I renamed the test file from ".patch"
to ".pl". I suppose I mistyped the extension when renumbering from 021
to 028.

> Did the DatumGetBool issue expose a deficiency in testing ?
> I think the !am_partition path was never being hit.

Hmm, the TAP test creates a subscription that contains both types of
tables.  I tried adding an assert for each case, and they were both hit
on running the test.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"La persona que no quería pecar / estaba obligada a sentarse
 en duras y empinadas sillas    / desprovistas, por cierto
 de blandos atenuantes"                          (Patricio Vogel)

  117. Re: Column Filtering in Logical Replication

    Justin Pryzby <pryzby@telsasoft.com> — 2022-01-03T14:40:57Z

    On Mon, Jan 03, 2022 at 11:31:39AM -0300, Alvaro Herrera wrote:
> > Did the DatumGetBool issue expose a deficiency in testing ?
> > I think the !am_partition path was never being hit.
> 
> Hmm, the TAP test creates a subscription that contains both types of
> tables.  I tried adding an assert for each case, and they were both hit
> on running the test.

Yes, I know both paths are hit now that it uses GetBool.

What I'm wondering is why tests didn't fail when one path wasn't hit - when it
said am_partition=DatumGetChar(); if (!am_partition){}

I suppose it's because the am_partition=true case correctly handles
nonpartitions.

Maybe the !am_partition case should be removed, and add a comment that
pg_partition_tree(pg_partition_root(%u))) also handles non-partitions.
Or maybe that's inefficient...

-- 
Justin

  118. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-01-03T15:01:59Z

    On 2022-Jan-03, Justin Pryzby wrote:

> Yes, I know both paths are hit now that it uses GetBool.
> 
> What I'm wondering is why tests didn't fail when one path wasn't hit - when it
> said am_partition=DatumGetChar(); if (!am_partition){}

Ah!

> I suppose it's because the am_partition=true case correctly handles
> nonpartitions.
> 
> Maybe the !am_partition case should be removed, and add a comment that
> pg_partition_tree(pg_partition_root(%u))) also handles non-partitions.
> Or maybe that's inefficient...

Hmm, that doesn't sound true.  Running the query manually, you get an
empty list if you use pg_partition_tree(pg_partition_root) with a
non-partition.  Maybe what was happening is that all columns were being
transmitted instead of only the required columns.  Maybe you're right
that the test isn't complete enough.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/

  119. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-01-06T08:37:33Z

    On Mon, Jan 3, 2022 at 8:01 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>

fetch_remote_table_info()
{
..
+ appendStringInfo(&cmd,
+ "  SELECT pg_catalog.unnest(prattrs)\n"
+ "    FROM pg_catalog.pg_publication p JOIN\n"
+ "         pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ "   WHERE p.pubname IN (%s) AND\n",
+ publications.data);
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ "    FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);

IIUC, this doesn't deal with cases when some publication has not
specified table attrs. In those cases, I think it should return all
attrs? Also, it is not very clear to me what exactly we want to do
with partitions?

-- 
With Regards,
Amit Kapila.

  120. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-01-06T09:41:30Z

    On Mon, Dec 27, 2021 at 10:36 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> Determining that an array has a NULL element seems convoluted.  I ended
> up with this query, where comparing the result of array_positions() with
> an empty array does that.  If anybody knows of a simpler way, or any
> situations in which this fails, I'm all ears.
>
> with published_cols as (
>         select case when
>                 pg_catalog.array_positions(pg_catalog.array_agg(unnest), null) <> '{}' then null else
>                 pg_catalog.array_agg(distinct unnest order by unnest) end AS attrs
>         from pg_catalog.pg_publication p join
>                 pg_catalog.pg_publication_rel pr on (p.oid = pr.prpubid) left join
>                 unnest(prattrs) on (true)
>         where prrelid = 38168 and p.pubname in ('pub1', 'pub2')
> )
> SELECT a.attname,
>        a.atttypid,
>        a.attnum = ANY(i.indkey)
>   FROM pg_catalog.pg_attribute a
>   LEFT JOIN pg_catalog.pg_index i
>        ON (i.indexrelid = pg_get_replica_identity_index(38168)),
>      published_cols
>  WHERE a.attnum > 0::pg_catalog.int2
>    AND NOT a.attisdropped and a.attgenerated = ''
>    AND a.attrelid = 38168
>    AND (published_cols.attrs IS NULL OR attnum = ANY(published_cols.attrs))
>  ORDER BY a.attnum;
>
> This returns all columns if at least one publication has a NULL prattrs,
> or only the union of columns listed in all publications, if all
> publications have a list of columns.
>

Considering this, don't we need to deal with "For All Tables" and "For
All Tables In Schema .." Publications in this query? The row filter
patch deal with such cases. The row filter patch handles the NULL case
via C code which makes the query relatively simpler. I am not sure if
the same logic can be used here but having a simple query here have
merit that if we want to use a single query to fetch both column and
row filters then we should be able to enhance it without making it
further complicated.

> (I was worried about obtaining the list of publications, but it turns
> out that it's already as a convenient list of OIDs in the MySubscription
> struct.)
>
> With this, we can remove the second query added by Rahila's original patch to
> filter out nonpublished columns.
>
> I still need to add pg_partition_tree() in order to search for
> publications containing a partition ancestor.  I'm not yet sure what
> happens (and what *should* happen) if an ancestor is part of a
> publication and the partition is also part of a publication, and the
> column lists differ.
>

Shouldn't we try to have a behavior similar to the row filter patch
for this case? The row filter patch behavior is as follows: "If your
publication contains a partitioned table, the publication parameter
publish_via_partition_root determines if it uses the partition row
filter (if the parameter is false, the default) or the root
partitioned table row filter. During initial tablesync, it doesn't do
any special handling for partitions.

-- 
With Regards,
Amit Kapila.

  121. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-01-06T12:22:54Z

    On 2022-Jan-06, Amit Kapila wrote:

> On Mon, Jan 3, 2022 at 8:01 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> >
> 
> fetch_remote_table_info()
> {
> ..
> + appendStringInfo(&cmd,
> + "  SELECT pg_catalog.unnest(prattrs)\n"
> + "    FROM pg_catalog.pg_publication p JOIN\n"
> + "         pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
> + "   WHERE p.pubname IN (%s) AND\n",
> + publications.data);
> + if (!am_partition)
> + appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
> + else
> + appendStringInfo(&cmd,
> + "prrelid IN (SELECT relid\n"
> + "    FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
> + lrel->remoteid);
> 
> IIUC, this doesn't deal with cases when some publication has not
> specified table attrs. In those cases, I think it should return all
> attrs?

Hmm, no, the idea here is that the list of columns should be null; the
code that uses this result is supposed to handle a null result to mean
hat all columns are included.

> Also, it is not very clear to me what exactly we want to do
> with partitions?

... Hmm, maybe there is a gap in testing here, I'll check; but the idea
is that we would use the column list of the most immediate ancestor that
has one, if the partition itself doesn't have one.  (I see we're missing
a check for "pubviaroot", which should represent an override.  Need more
tests here.)

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Uno puede defenderse de los ataques; contra los elogios se esta indefenso"

  122. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-01-06T12:29:31Z

    On 2022-Jan-06, Amit Kapila wrote:

> Considering this, don't we need to deal with "For All Tables" and "For
> All Tables In Schema .." Publications in this query? The row filter
> patch deal with such cases. The row filter patch handles the NULL case
> via C code which makes the query relatively simpler.

Yes.  I realized after sending that email that the need to handle schema
publications would make a single query very difficult, so I ended up
splitting it again in two queries, which is what you see in the latest
version submitted.

> I am not sure if the same logic can be used here but having a simple
> query here have merit that if we want to use a single query to fetch
> both column and row filters then we should be able to enhance it
> without making it further complicated.

I have looked the row filter code a couple of times to make sure we're
somewhat compatible, but didn't look closely enough to see if we can
make the queries added by both patches into a single one.

> Shouldn't we try to have a behavior similar to the row filter patch
> for this case? The row filter patch behavior is as follows: "If your
> publication contains a partitioned table, the publication parameter
> publish_via_partition_root determines if it uses the partition row
> filter (if the parameter is false, the default) or the root
> partitioned table row filter. During initial tablesync, it doesn't do
> any special handling for partitions.

I'll have a look.

Thanks for looking!

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/

  123. Re: Column Filtering in Logical Replication

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-01-07T11:46:35Z

    I think this is getting pretty good now.  I like the overall behavior now.

Some details:

There are still a few references to "filter", but I see most of the
patch now uses column list or something.  Maybe do another cleanup
pass before finalizing the patch.

doc/src/sgml/catalogs.sgml needs to be updated.

doc/src/sgml/ref/alter_publication.sgml:

"allows to change" -> "allows changing"

src/backend/catalog/pg_publication.c:

publication_translate_columns(): I find the style of having a couple
of output arguments plus a return value that is actually another
output value confusing.  (It would be different if the return value
was some kind of success value.)  Let's make it all output arguments.

About the XXX question there: I would make the column numbers always
sorted.  I don't have a strong reason for this, but otherwise we might
get version differences, unstable dumps etc.  It doesn't seem
complicated to keep this a bit cleaner.

I think publication_translate_columns() also needs to prohibit
generated columns.  We already exclude those implicitly throughout the
logical replication code, but if a user explicitly set one here,
things would probably break.

src/backend/commands/tablecmds.c:

ATExecReplicaIdentity(): Regarding the question of how to handle
REPLICA_IDENTITY_NOTHING: I see two ways to do this.  Right now, the
approach is that the user can set the replica identity freely, and we
decide later based on that what we can replicate (e.g., no updates).
For this patch, that would mean we don't restrict what columns can be
in the column list, but we check what actions we can replicate based
on the column list.  The alternative is that we require the column
list to include the replica identity, as the patch is currently doing,
which would mean that REPLICA_IDENTITY_NOTHING can be allowed since
it's essentially a set of zero columns.

I find the current behavior a bit weird on reflection.  If a user
wants to replicate on some columns and only INSERTs, that should be
allowed regardless of what the replica identity columns are.

src/backend/replication/pgoutput/pgoutput.c:

In get_rel_sync_entry(), why did you remove the block

-           if (entry->pubactions.pubinsert && 
entry->pubactions.pubupdate &&
-               entry->pubactions.pubdelete && 
entry->pubactions.pubtruncate)
-               break;

Maybe this is intentional, but it's not clear to me.

  124. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-01-07T13:08:58Z

    On Fri, Jan 7, 2022 at 5:16 PM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:
>
> src/backend/commands/tablecmds.c:
>
> ATExecReplicaIdentity(): Regarding the question of how to handle
> REPLICA_IDENTITY_NOTHING: I see two ways to do this.  Right now, the
> approach is that the user can set the replica identity freely, and we
> decide later based on that what we can replicate (e.g., no updates).
>

+1. This is what we are trying to do with the row filter patch. It
seems Hou-San has also mentioned the same on this thread [1].

> For this patch, that would mean we don't restrict what columns can be
> in the column list, but we check what actions we can replicate based
> on the column list.  The alternative is that we require the column
> list to include the replica identity, as the patch is currently doing,
> which would mean that REPLICA_IDENTITY_NOTHING can be allowed since
> it's essentially a set of zero columns.
>
> I find the current behavior a bit weird on reflection.  If a user
> wants to replicate on some columns and only INSERTs, that should be
> allowed regardless of what the replica identity columns are.
>

Right, I also raised the same point [2] related to INSERTs.

[1] - https://www.postgresql.org/message-id/OS0PR01MB5716330FFE3803DF887D073C94789%40OS0PR01MB5716.jpnprd01.prod.outlook.com
[2] - https://www.postgresql.org/message-id/CAA4eK1%2BFoJ-J7wUG5s8zCtY0iBuN9LcjQcYhV4BD17xhuHfoug%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  125. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-01-10T21:38:40Z

    In this version I have addressed these points, except the REPLICA
IDENTITY NOTHING stuff.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"The eagle never lost so much time, as
when he submitted to learn of the crow." (William Blake)

  126. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-01-11T00:28:48Z

    On 2022-Jan-07, Peter Eisentraut wrote:

> ATExecReplicaIdentity(): Regarding the question of how to handle
> REPLICA_IDENTITY_NOTHING: I see two ways to do this.  Right now, the
> approach is that the user can set the replica identity freely, and we
> decide later based on that what we can replicate (e.g., no updates).
> For this patch, that would mean we don't restrict what columns can be
> in the column list, but we check what actions we can replicate based
> on the column list.  The alternative is that we require the column
> list to include the replica identity, as the patch is currently doing,
> which would mean that REPLICA_IDENTITY_NOTHING can be allowed since
> it's essentially a set of zero columns.
> 
> I find the current behavior a bit weird on reflection.  If a user
> wants to replicate on some columns and only INSERTs, that should be
> allowed regardless of what the replica identity columns are.

Hmm.  So you're saying that we should only raise errors about the column
list if we are publishing UPDATE or DELETE, but otherwise let the
replica identity be anything.  OK, I'll see if I can come up with a
reasonable set of rules ...

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Before you were born your parents weren't as boring as they are now. They
got that way paying your bills, cleaning up your room and listening to you
tell them how idealistic you are."  -- Charles J. Sykes' advice to teenagers

  127. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-01-11T18:52:22Z

    On 2022-Jan-10, Alvaro Herrera wrote:

> Hmm.  So you're saying that we should only raise errors about the column
> list if we are publishing UPDATE or DELETE, but otherwise let the
> replica identity be anything.  OK, I'll see if I can come up with a
> reasonable set of rules ...

This is an attempt to do it that way.  Now you can add a table to a
publication without regards for how column filter compares to the
replica identity, as long as the publication does not include updates
and inserts.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
"La fuerza no está en los medios físicos
sino que reside en una voluntad indomable" (Gandhi)

  128. Re: Column Filtering in Logical Replication

    Justin Pryzby <pryzby@telsasoft.com> — 2022-01-11T21:10:53Z

    Is there any coordination between the "column filter" patch and the "row
filter" patch ?  Are they both on track for PG15 ?  Has anybody run them
together ?

Whichever patch is merged 2nd should include tests involving a subset of
columns along with a WHERE clause.

I have a suggestion: for the functions for which both patches are adding
additional argument types, define a filtering structure for both patches to
use.  Similar to what we did for some utility statements in a3dc92600.

I'm referring to:
logicalrep_write_update()
logicalrep_write_tuple()

That would avoid avoid some rebase conflicts on april 9, and avoid functions
with 7,8,9 arguments, and maybe simplify adding arguments in the future.

-- 
Justin

  129. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-01-12T00:41:28Z

    On 2022-Jan-11, Alvaro Herrera wrote:

> On 2022-Jan-10, Alvaro Herrera wrote:
> 
> > Hmm.  So you're saying that we should only raise errors about the column
> > list if we are publishing UPDATE or DELETE, but otherwise let the
> > replica identity be anything.  OK, I'll see if I can come up with a
> > reasonable set of rules ...
> 
> This is an attempt to do it that way.  Now you can add a table to a
> publication without regards for how column filter compares to the
> replica identity, as long as the publication does not include updates
> and inserts.

I discovered a big hole in this, which is that ALTER PUBLICATION SET
(publish='insert,update') can add UPDATE publishing to a publication
that was only publishing INSERTs.  It's easy to implement a fix: in
AlterPublicationOptions, scan the list of tables and raise an error if
any of them has a column list that doesn't include all the columns in
the replica identity.

However, that proposal has an ugly flaw: there is no index on
pg_publication_rel.prpubid, which means that the only way to find the
relations we need to inspect is to seqscan pg_publication_rel.

Also, psql's query for \dRp+ uses a seqscan in pg_publication_rel.

Therefore, I propose to add an index on pg_publication_rel.prpubid.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
"¿Qué importan los años?  Lo que realmente importa es comprobar que
a fin de cuentas la mejor edad de la vida es estar vivo"  (Mafalda)

  130. Re: Column Filtering in Logical Replication

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-01-12T11:30:13Z

    On 12.01.22 01:41, Alvaro Herrera wrote:
> Therefore, I propose to add an index on pg_publication_rel.prpubid.

That seems very reasonable.

  131. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-01-12T18:57:12Z

    On 2022-Jan-11, Justin Pryzby wrote:

> Is there any coordination between the "column filter" patch and the "row
> filter" patch ?

Not beyond the grammar, which I tested.

> Are they both on track for PG15 ?

I think they're both on track, yes.

> Has anybody run them together ?

Not me.

> I have a suggestion: for the functions for which both patches are adding
> additional argument types, define a filtering structure for both patches to
> use.  Similar to what we did for some utility statements in a3dc92600.
> 
> I'm referring to:
> logicalrep_write_update()
> logicalrep_write_tuple()

Fixed: the row filter patch no longer adds extra arguments to those
functions.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"Tiene valor aquel que admite que es un cobarde" (Fernandel)

  132. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-01-14T11:51:41Z

    On Wed, Jan 12, 2022 at 2:40 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> Is there any coordination between the "column filter" patch and the "row
> filter" patch ?  Are they both on track for PG15 ?  Has anybody run them
> together ?
>

The few things where I think we might need to define some common
behavior are as follows:

1. Replica Identity handling: Currently the column filter patch gives
an error during create/alter subscription if the specified column list
is invalid (Replica Identity columns are missing). It also gives an
error if the user tries to change the replica identity. However, it
doesn't deal with cases where the user drops and adds a different
primary key that has a different set of columns which can lead to
failure during apply on the subscriber.

I think another issue w.r.t column filter patch is that even while
creating publication (even for 'insert' publications) it should check
that all primary key columns must be part of published columns,
otherwise, it can fail while applying on subscriber as it will try to
insert NULL for the primary key column.

2. Handling of partitioned tables vs. Replica Identity (RI): When
adding a partitioned table with a column list to the publication (with
publish_via_partition_root = false), we should check the Replica
Identity of all its leaf partition as the RI on the partition is the
one actually takes effect when publishing DML changes. We need to
check RI while attaching the partition as well, as the newly added
partitions will automatically become part of publication if the
partitioned table is part of the publication. If we don't do this the
later deletes/updates can fail.

All these cases are dealt with in row filter patch because of the
on-the-fly check which means we check the validation of columns in row
filters while actual operation update/delete via
CheckCmdReplicaIdentity and cache the result of same for future use.
This is inline with existing checks of RI vs. operations on tables.
The primary reason for this was we didn't want to handle validation of
row filters at so many places.

3. Tablesync.c handling: Ideally, it would be good if we have a single
query to fetch both row filters and column filters but even if that is
not possible in the first version, the behavior should be same for
both queries w.r.t partitioned tables, For ALL Tables and For All
Tables In Schema cases.

Currently, the column filter patch doesn't seem to respect For ALL
Tables and For All Tables In Schema cases, basically, it just copies
the columns it finds through some of the publications even if one of
the publications is defined as For All Tables. The row filter patch
ignores the row filters if one of the publications is defined as For
ALL Tables and For All Tables In Schema.

For row filter patch, if the publication contains a partitioned table,
the publication parameter publish_via_partition_root determines if it
uses the partition row filter (if the parameter is false, the default)
or the root partitioned table row filter and this is taken care of
even during the initial tablesync.

For column filter patch, if the publication contains a partitioned
table, it seems that it finds all columns that the tables in its
partition tree specified in the publications, whether
publish_via_partition_root is true or false.

We have done some testing w.r.t above cases with both patches and my
colleague will share the results.

-- 
With Regards,
Amit Kapila.

  133. RE: Column Filtering in Logical Replication

    tanghy <tanghy.fnst@fujitsu.com> — 2022-01-14T13:18:49Z

    On Friday, January 14, 2022 7:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> 
> On Wed, Jan 12, 2022 at 2:40 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> >
> > Is there any coordination between the "column filter" patch and the "row
> > filter" patch ?  Are they both on track for PG15 ?  Has anybody run them
> > together ?
> >
> 
> The few things where I think we might need to define some common
> behavior are as follows:
> 

I tried some cases about the points you mentions, which can be taken as
reference.

> 1. Replica Identity handling: Currently the column filter patch gives
> an error during create/alter subscription if the specified column list
> is invalid (Replica Identity columns are missing). It also gives an
> error if the user tries to change the replica identity. However, it
> doesn't deal with cases where the user drops and adds a different
> primary key that has a different set of columns which can lead to
> failure during apply on the subscriber.
> 

An example for this scenario:
-- publisher --
create table tbl(a int primary key, b int);
create publication pub for table tbl(a);
alter table tbl drop CONSTRAINT tbl_pkey;
alter table tbl add primary key (b);

-- subscriber --
create table tbl(a int, b int);
create subscription sub connection 'port=5432 dbname=postgres' publication pub;

-- publisher --
insert into tbl values (1,1);

-- subscriber --
postgres=# select * from tbl;
 a | b
---+---
 1 |
(1 row)

update tbl set b=1 where a=1;
alter table tbl add primary key (b);

-- publisher --
delete from tbl;


The subscriber reported the following error message and DELETE failed in subscriber.
ERROR:  publisher did not send replica identity column expected by the logical replication target relation "public.tbl"
CONTEXT:  processing remote data during "DELETE" for replication target relation "public.tbl" in transaction 723 at 2022-01-14 13:11:51.514261+08

-- subscriber
postgres=# select * from tbl;
 a | b
---+---
 1 | 1
(1 row)

> I think another issue w.r.t column filter patch is that even while
> creating publication (even for 'insert' publications) it should check
> that all primary key columns must be part of published columns,
> otherwise, it can fail while applying on subscriber as it will try to
> insert NULL for the primary key column.
> 

For example:
-- publisher --
create table tbl(a int primary key, b int);
create publication pub for table tbl(a);
alter table tbl drop CONSTRAINT tbl_pkey;
alter table tbl add primary key (b);

-- subscriber --
create table tbl(a int, b int primary key);
create subscription sub connection 'port=5432 dbname=postgres' publication pub;

-- publisher --
insert into tbl values (1,1);

The subscriber reported the following error message and INSERT failed in subscriber.
ERROR:  null value in column "b" of relation "tbl" violates not-null constraint
DETAIL:  Failing row contains (1, null).

-- subscriber --
postgres=# select * from tbl;
 a | b
---+---
(0 rows)

> 2. Handling of partitioned tables vs. Replica Identity (RI): When
> adding a partitioned table with a column list to the publication (with
> publish_via_partition_root = false), we should check the Replica
> Identity of all its leaf partition as the RI on the partition is the
> one actually takes effect when publishing DML changes. We need to
> check RI while attaching the partition as well, as the newly added
> partitions will automatically become part of publication if the
> partitioned table is part of the publication. If we don't do this the
> later deletes/updates can fail.
> 

Please see the following 3 cases about partition.

Case1 (publish a parent table which has a partition table):
----------------------------
-- publisher --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create unique INDEX ON child (a,b);
alter table child alter a set not null;
alter table child alter b set not null;
alter table child replica identity using INDEX child_a_b_idx;
create publication pub for table parent(a) with(publish_via_partition_root=false);

-- subscriber --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres' publication pub;

-- publisher --
insert into parent values (1,1);

-- subscriber --
postgres=# select * from parent;
 a | b
---+---
 1 |
(1 row)

-- add RI in subscriber to avoid other errors
update child set b=1 where a=1;
create unique INDEX ON child (a,b);
alter table child alter a set not null;
alter table child alter b set not null;
alter table child replica identity using INDEX child_a_b_idx;

-- publisher --
delete from parent;

The subscriber reported the following error message and DELETE failed in subscriber.
ERROR:  publisher did not send replica identity column expected by the logical replication target relation "public.child"
CONTEXT:  processing remote data during "DELETE" for replication target relation "public.child" in transaction 727 at 2022-01-14 20:29:46.50784+08

-- subscriber --
postgres=# select * from parent;
 a | b
---+---
 1 | 1
(1 row)


Case2 (create publication for parent table, then alter table to attach partition):
----------------------------
-- publisher --
create table parent (a int, b int) partition by range (a);
create table child (a int, b int);
create unique INDEX ON child (a,b);
alter table child alter a set not null;
alter table child alter b set not null;
alter table child replica identity using INDEX child_a_b_idx;
create publication pub for table parent(a) with(publish_via_partition_root=false);
alter table parent attach partition child default;
insert into parent values (1,1);

-- subscriber --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres' publication pub;

postgres=# select * from parent;
 a | b
---+---
 1 |
(1 row)

-- add RI in subscriber to avoid other errors
update child set b=1 where a=1;
create unique INDEX ON child (a,b);
alter table child alter a set not null;
alter table child alter b set not null;
alter table child replica identity using INDEX child_a_b_idx;

-- publisher --
delete from parent;

The subscriber reported the following error message and DELETE failed in subscriber.
ERROR:  publisher did not send replica identity column expected by the logical replication target relation "public.child"
CONTEXT:  processing remote data during "DELETE" for replication target relation "public.child" in transaction 728 at 2022-01-14 20:42:16.483878+08

-- subscriber --
postgres=# select * from parent;
 a | b
---+---
 1 | 1
(1 row)


Case3 (create publication for parent table, then using "create table partition
of", and specify primary key when creating partition table):
----------------------------
-- publisher --
create table parent (a int, b int) partition by range (a);
create publication pub for table parent(a) with(publish_via_partition_root=false);
create table child partition of parent (primary key (a,b)) default;

-- subscriber --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres' publication pub;

-- publisher --
insert into parent values (1,1);

-- subscriber --
postgres=# select * from parent;
 a | b
---+---
 1 |
(1 row)

-- add PK in subscriber to avoid other errors
update child set b=1 where a=1;
alter table child add primary key (a,b);

-- publisher --
delete from parent;

The subscriber reported the following error message and DELETE failed in subscriber.
ERROR:  publisher did not send replica identity column expected by the logical replication target relation "public.child"
CONTEXT:  processing remote data during "DELETE" for replication target relation "public.child" in transaction 723 at 2022-01-14 20:45:33.622168+08

-- subscriber --
postgres=# select * from parent;
 a | b
---+---
 1 | 1
(1 row)

> 3. Tablesync.c handling: Ideally, it would be good if we have a single
> query to fetch both row filters and column filters but even if that is
> not possible in the first version, the behavior should be same for
> both queries w.r.t partitioned tables, For ALL Tables and For All
> Tables In Schema cases.
> 
> Currently, the column filter patch doesn't seem to respect For ALL
> Tables and For All Tables In Schema cases, basically, it just copies
> the columns it finds through some of the publications even if one of
> the publications is defined as For All Tables. The row filter patch
> ignores the row filters if one of the publications is defined as For
> ALL Tables and For All Tables In Schema.
> 

A case for the publications is defined as For ALL Tables and For All Tables In
Schema:
-- publisher --
create schema s1;
create table s1.t1 (a int, b int);
create publication p1 for table s1.t1 (a);
create publication p2 for all tables;
insert into s1.t1 values (1,1);

-- subscriber --
create schema s1;
create table s1.t1 (a int, b int);
create subscription sub connection 'port=5432 dbname=postgres' publication p1, p2;
postgres=# select * from s1.t1;
 a | b
---+---
 1 |
(1 row)

(I got the same result when p2 is specified as "FOR ALL TABLES IN SCHEMA s1")

> For row filter patch, if the publication contains a partitioned table,
> the publication parameter publish_via_partition_root determines if it
> uses the partition row filter (if the parameter is false, the default)
> or the root partitioned table row filter and this is taken care of
> even during the initial tablesync.
> 
> For column filter patch, if the publication contains a partitioned
> table, it seems that it finds all columns that the tables in its
> partition tree specified in the publications, whether
> publish_via_partition_root is true or false.
> 

Please see the following cases.

Column filter
----------------------------------------
-- publisher --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create publication p1 for table parent (a) with(publish_via_partition_root=false);
create publication p2 for table parent (a) with(publish_via_partition_root=true);
insert into parent values (1,1);

-- subscriber --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres' publication p1;
postgres=# select * from parent;  -- column filter works when publish_via_partition_root=false
 a | b
---+---
 1 |
(1 row)

drop subscription sub;
delete from parent;
create subscription sub connection 'port=5432 dbname=postgres' publication p2;
postgres=# select * from parent;  -- column filter also works when publish_via_partition_root=true
 a | b
---+---
 1 |
(1 row)


Row filter
----------------------------------------
-- publisher --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create publication p1 for table parent where (a>10) with(publish_via_partition_root=false);
create publication p2 for table parent where (a>10) with(publish_via_partition_root=true);
insert into parent values (1,1);
insert into parent values (11,11);

-- subscriber
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres' publication p1;
postgres=# select * from parent;  -- row filter doesn't work when publish_via_partition_root=false
 a  | b
----+----
  1 |  1
 11 | 11
(2 rows)

drop subscription sub;
delete from parent;
create subscription sub connection 'port=5432 dbname=postgres' publication p2;
postgres=# select * from parent;  -- row filter works when publish_via_partition_root=true
 a  | b
----+----
 11 | 11
(1 row)

Regards,
Tang

  134. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-01-14T13:38:00Z

    On 2022-Jan-14, Amit Kapila wrote:

> 1. Replica Identity handling: Currently the column filter patch gives
> an error during create/alter subscription if the specified column list
> is invalid (Replica Identity columns are missing). It also gives an
> error if the user tries to change the replica identity. However, it
> doesn't deal with cases where the user drops and adds a different
> primary key that has a different set of columns which can lead to
> failure during apply on the subscriber.

Hmm, yeah, I suppose we should check that the primary key is compatible
with the column list in all publications.  (I wonder what happens in the
interim, that is, what happens to tuples modified after the initial PK
is dropped and before the new PK is installed.  Are these considered to
have "replica identiy nothing"?)

> I think another issue w.r.t column filter patch is that even while
> creating publication (even for 'insert' publications) it should check
> that all primary key columns must be part of published columns,
> otherwise, it can fail while applying on subscriber as it will try to
> insert NULL for the primary key column.

I'm not so sure about the primary key aspects, actually; keep in mind
that the replica can have a different table definition, and it might
have even a completely different primary key.  I think this part is up
to the user to set up correctly; we have enough with just trying to make
the replica identity correct.

> 2. Handling of partitioned tables vs. Replica Identity (RI): When
> adding a partitioned table with a column list to the publication (with
> publish_via_partition_root = false), we should check the Replica
> Identity of all its leaf partition as the RI on the partition is the
> one actually takes effect when publishing DML changes. We need to
> check RI while attaching the partition as well, as the newly added
> partitions will automatically become part of publication if the
> partitioned table is part of the publication. If we don't do this the
> later deletes/updates can fail.

Hmm, yeah.

> 3. Tablesync.c handling: Ideally, it would be good if we have a single
> query to fetch both row filters and column filters but even if that is
> not possible in the first version, the behavior should be same for
> both queries w.r.t partitioned tables, For ALL Tables and For All
> Tables In Schema cases.
> 
> Currently, the column filter patch doesn't seem to respect For ALL
> Tables and For All Tables In Schema cases, basically, it just copies
> the columns it finds through some of the publications even if one of
> the publications is defined as For All Tables. The row filter patch
> ignores the row filters if one of the publications is defined as For
> ALL Tables and For All Tables In Schema.

Oh, yeah, if a table appears in two publications and one of them is ALL
TABLES [IN SCHEMA], then we don't consider it as an all-columns
publication.  You're right, that should be corrected.

> We have done some testing w.r.t above cases with both patches and my
> colleague will share the results.

Great, thanks.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/

  135. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-01-15T03:45:07Z

    On Fri, Jan 14, 2022 at 7:08 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2022-Jan-14, Amit Kapila wrote:
>
> > 1. Replica Identity handling: Currently the column filter patch gives
> > an error during create/alter subscription if the specified column list
> > is invalid (Replica Identity columns are missing). It also gives an
> > error if the user tries to change the replica identity. However, it
> > doesn't deal with cases where the user drops and adds a different
> > primary key that has a different set of columns which can lead to
> > failure during apply on the subscriber.
>
> Hmm, yeah, I suppose we should check that the primary key is compatible
> with the column list in all publications.  (I wonder what happens in the
> interim, that is, what happens to tuples modified after the initial PK
> is dropped and before the new PK is installed.  Are these considered to
> have "replica identiy nothing"?)
>

I think so.

> > I think another issue w.r.t column filter patch is that even while
> > creating publication (even for 'insert' publications) it should check
> > that all primary key columns must be part of published columns,
> > otherwise, it can fail while applying on subscriber as it will try to
> > insert NULL for the primary key column.
>
> I'm not so sure about the primary key aspects, actually; keep in mind
> that the replica can have a different table definition, and it might
> have even a completely different primary key.  I think this part is up
> to the user to set up correctly; we have enough with just trying to make
> the replica identity correct.
>

But OTOH, the primary key is also considered default replica identity,
so I think users will expect it to work. You are right this problem
can also happen if the user defined a different primary key on a
replica but that is even a problem in HEAD (simple inserts will fail)
but I am worried about the case where both the publisher and
subscriber have the same primary key as that works in HEAD.

-- 
With Regards,
Amit Kapila.

  136. Re: Column Filtering in Logical Replication

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-01-18T10:33:19Z

    On 15.01.22 04:45, Amit Kapila wrote:
>>> I think another issue w.r.t column filter patch is that even while
>>> creating publication (even for 'insert' publications) it should check
>>> that all primary key columns must be part of published columns,
>>> otherwise, it can fail while applying on subscriber as it will try to
>>> insert NULL for the primary key column.
>>
>> I'm not so sure about the primary key aspects, actually; keep in mind
>> that the replica can have a different table definition, and it might
>> have even a completely different primary key.  I think this part is up
>> to the user to set up correctly; we have enough with just trying to make
>> the replica identity correct.
> 
> But OTOH, the primary key is also considered default replica identity,
> so I think users will expect it to work. You are right this problem
> can also happen if the user defined a different primary key on a
> replica but that is even a problem in HEAD (simple inserts will fail)
> but I am worried about the case where both the publisher and
> subscriber have the same primary key as that works in HEAD.

This would seem to be a departure from the current design of logical 
replication.  It's up to the user to arrange things so that data can be 
applied in general.  Otherwise, if the default assumption is that the 
schema is the same on both sides, then column filtering shouldn't exist 
at all, since that will necessarily break that assumption.

Maybe there could be a strict mode or something that has more checks, 
but that would be a separate feature.  The existing behavior is that you 
can publish anything you want and it's up to you to make sure the 
receiving side can store it.

  137. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-01-28T08:39:41Z

    Here are some review comments for the v17-0001 patch.

~~~

1. Commit message

If no column list is specified, all the columns are replicated, as
previously

Missing period (.) at the end of that sentence.

~~~

2. doc/src/sgml/catalogs.sgml

+      <para>
+       This is an array of values that indicates which table columns are
+       part of the publication.  For example a value of <literal>1 3</literal>
+       would mean that the first and the third table columns are published.
+       A null value indicates that all attributes are published.
+      </para></entry>

Missing comma:
"For example" --> "For example,"

Terms:
The text seems to jump between "columns" and "attributes". Perhaps,
for consistency, that last sentence should say: "A null value
indicates that all columns are published."

~~~

3. doc/src/sgml/protocol.sgml

 </variablelist>
-        Next, the following message part appears for each column
(except generated columns):
+        Next, the following message part appears for each column (except
+        generated columns and other columns that don't appear in the column
+        filter list, for tables that have one):
 <variablelist>

Perhaps that can be expressed more simply, like:

Next, the following message part appears for each column (except
generated columns and other columns not present in the optional column
filter list):

~~~

4. doc/src/sgml/ref/alter_publication.sgml

+ALTER PUBLICATION <replaceable class="parameter">name</replaceable>
ALTER TABLE <replaceable
class="parameter">publication_object</replaceable> SET COLUMNS { (
<replaceable class="parameter">name</replaceable> [, ...] ) | ALL }

The syntax chart looks strange because there is already a "TABLE" and
a column_name list within the "publication_object" definition, so do
ALTER TABLE and publication_object co-exist?
According to the current documentation it suggests nonsense like below is valid:
ALTER PUBLICATION mypublication ALTER TABLE TABLE t1 (a,b,c) SET
COLUMNS (a,b,c);

--

But more fundamentally, I don't see why any new syntax is even needed at all.

Instead of:
ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS
(user_id, firstname, lastname);
Why not just:
ALTER PUBLICATION mypublication ALTER TABLE users (user_id, firstname,
lastname);

Then, if the altered table defines a *different* column list then it
would be functionally equivalent to whatever your SET COLUMNS is doing
now. AFAIK this is how the Row-Filter [1] works, so that altering an
existing table to have a different Row-Filter just overwrites that
table's filter. IMO the Col-Filter behaviour should work the same as
that - "SET COLUMNS" is redundant.

~~~

5. doc/src/sgml/ref/alter_publication.sgml

-    TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [, ... ]
+    TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [ ( <replaceable
class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]

That extra comma after the "column_name" seems wrong because there is
one already in "[, ... ]".

~~~

6. doc/src/sgml/ref/create_publication.sgml

-    TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [, ... ]
+    TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [ ( <replaceable
class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]

(Same as comment #5).
That extra comma after the "column_name" seems wrong because there is
one already in "[, ... ]".

~~~

7. doc/src/sgml/ref/create_publication.sgml

+     <para>
+      When a column list is specified, only the listed columns are replicated;
+      any other columns are ignored for the purpose of replication through
+      this publication.  If no column list is specified, all columns of the
+      table are replicated through this publication, including any columns
+      added later.  If a column list is specified, it must include the replica
+      identity columns.
+     </para>

Suggest to re-word this a bit simpler:

e.g.
- "listed columns" --> "named columns"
- I don't think it is necessary to say the unlisted columns are ignored.
- I didn't think it is necessary to say "though this publication"

AFTER
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated,
including any columns added later. If a column list is specified, it must
include the replica identity columns.

~~~

8. doc/src/sgml/ref/create_publication.sgml

Consider adding another example showing a CREATE PUBLICATION which has
a column list.

~~~

9. src/backend/catalog/pg_publication.c - check_publication_add_relation

 /*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of attribute numbers given in the column list,
+ * or NULL if it was not specified.
  */

Typo: "targetcols" --> "columns" ??

~~~

10. src/backend/catalog/pg_publication.c - check_publication_add_relation

+
+ /* Make sure the column list checks out */
+ if (columns != NULL)
+ {

Perhaps "checks out" could be worded better.

~~~

11. src/backend/catalog/pg_publication.c - check_publication_add_relation

+ /* Make sure the column list checks out */
+ if (columns != NULL)
+ {
+ /*
+ * Even if the user listed all columns in the column list, we cannot
+ * allow a column list to be specified when REPLICA IDENTITY is FULL;
+ * that would cause problems if a new column is added later, because
+ * the new column would have to be included (because of being part of
+ * the replica identity) but it's technically not allowed (because of
+ * not being in the publication's column list yet).  So reject this
+ * case altogether.
+ */
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+    RelationGetRelationName(targetrel)),
+ errdetail("Cannot specify a column list on relations with REPLICA
IDENTITY FULL."));
+
+ check_publication_columns(pub, targetrel, columns);
+ }

IIUC almost all of the above comment and code is redundant because by
calling the check_publication_columns function it will do exactly the
same check...

So,  that entire slab might be replaced by 2 lines:

if (columns != NULL)
check_publication_columns(pub, targetrel, columns);

~~~

12. src/backend/catalog/pg_publication.c - publication_set_table_columns

+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+   Relation targetrel, List *columns)
+{
+ Bitmapset  *attset;
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));

It seemed curious to use memset false for "replaces" but memset 0 for
"nulls", since they are both bool arrays (??)

~~~

13. src/backend/catalog/pg_publication.c - compare_int16

+/* qsort comparator for attnums */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}

This comparator seems common with another one already in the PG
source. Perhaps it would be better for generic comparators (like this
one) to be in some common code instead of scattered cut/paste copies
of the same thing.

~~~

14. src/backend/commands/publicationcmds.c - AlterPublicationTables

+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+   linitial_node(PublicationTable, tables));
+ }

(Same as my earlier review comment #4)

Suggest to call this PublicationSetColumns based on some smarter
detection logic of a changed column list. Please refer to the
Row-Filter patch [1] for this same function.

~~~

15. src/backend/commands/publicationcmds.c - AlterPublicationTables

+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;

Perhaps a more explanatory comment would be better there?

~~~

16. src/backend/commands/tablecmds.c - relation_mark_replica_identity

@@ -15841,6 +15871,7 @@ relation_mark_replica_identity(Relation rel,
char ri_type, Oid indexOid,
  CatalogTupleUpdate(pg_index, &pg_index_tuple->t_self, pg_index_tuple);
  InvokeObjectPostAlterHookArg(IndexRelationId, thisIndexOid, 0,
  InvalidOid, is_internal);
+
  /*
  * Invalidate the relcache for the table, so that after we commit
  * all sessions will refresh the table's replica identity index

Spurious whitespace change seemed unrelated to the Col-Filter patch.

~~~

17. src/backend/parser/gram.y

  *
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *

(Same as my earlier review comment #4)

IMO there was no need for the new syntax of SET COLUMNS.

~~~

18. src/backend/replication/logical/proto.c - logicalrep_write_attrs

- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc,
i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
  /* fetch bitmap of REPLICATION IDENTITY attributes */
  replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
  if (!replidentfull)
  idattrs = RelationGetIdentityKeyBitmap(rel);

+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+

This change seemed to have the effect of moving that 4 lines of
"replidentfull" code from below the loop to above the loop. But moving
that code seems unrelated to the Col-Filter patch. (??).

~~~

19. src/backend/replication/logical/tablesync.c - fetch_remote_table_info

@@ -793,12 +877,12 @@ fetch_remote_table_info(char *nspname, char *relname,

  ExecClearTuple(slot);
  }
+
  ExecDropSingleTupleTableSlot(slot);
-
- lrel->natts = natt;
-
  walrcv_clear_result(res);
  pfree(cmd.data);
+
+ lrel->natts = natt;
 }

The shuffling of those few lines seems unrelated to any requirement of
the Col-Filter patch (??)

~~~

20. src/backend/replication/logical/tablesync.c - copy_table

+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }

Perhaps that could be expressed more simply if the other way around like:

for (int i = 0; i < lrel.natts; i++)
{
if (i)
appendStringInfoString(&cmd, ", ");
appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
}

~~~

21. src/backend/replication/pgoutput/pgoutput.c

+
+ /*
+ * Set of columns included in the publication, or NULL if all columns are
+ * included implicitly.  Note that the attnums in this list are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset  *columns;

Typo: "in this list" --> "in this set" (??)

~~~

22. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry

  * Don't publish changes for partitioned tables, because
- * publishing those of its partitions suffices, unless partition
- * changes won't be published due to pubviaroot being set.
+ * publishing those of its partitions suffices.  (However, ignore
+ * this if partition changes are not to published due to
+ * pubviaroot being set.)
  */

This change seems unrelated to the Col-Filter patch, so perhaps it
should not be here at all.

Also, typo: "are not to published"

~~~

23. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry

+ /*
+ * Obtain columns published by this publication, and add them
+ * to the list for this rel.  Note that if at least one
+ * publication has a empty column list, that means to publish
+ * everything; so if we saw a publication that includes all
+ * columns, skip this.
+ */

Typo: "a empty" --> "an empty"

~~~

24. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry

+ if (isnull)
+ {
+ /*
+ * If we see a publication with no columns, reset the
+ * list and ignore further ones.
+ */

Perhaps that comment is meant to say "with no column filter" instead
of "with no columns"?

~~~

25. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry

+ if (isnull)
+ {
...
+ }
+ else if (!isnull)
+ {
...
+ }

Is the "if (!isnull)" in the else just to be really REALLY sure it is not null?

~~~

26. src/bin/pg_dump/pg_dump.c - getPublicationTables

+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;

I got confused reading this code. Are those different indices 'i' and
'j' correct?

~~~

27. src/bin/psql/describe.c

The Row-Filter [1] displays filter information not only for the psql
\dRp+ command but also for the psql \d <tablename> command. Perhaps
the Col-Filter patch should do that too.

~~~

28. src/bin/psql/tab-complete.c

@@ -1657,6 +1657,8 @@ psql_completion(const char *text, int start, int end)
  /* ALTER PUBLICATION <name> ADD */
  else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
  COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
  /* ALTER PUBLICATION <name> DROP */

I am not sure about this one- is that change even related to the
Col-Filter patch or is this some unrelated bugfix?

~~~

29. src/include/catalog/pg_publication.h

@@ -86,6 +86,7 @@ typedef struct Publication
 typedef struct PublicationRelInfo
 {
  Relation relation;
+ List    *columns;
 } PublicationRelInfo;

Perhaps that needs some comment. e.g. do you need to mention that a
NIL List means all columns?

~~~

30. src/include/nodes/parsenodes.h

@@ -3642,6 +3642,7 @@ typedef struct PublicationTable
 {
  NodeTag type;
  RangeVar   *relation; /* relation to be published */
+ List    *columns; /* List of columns in a publication table */
 } PublicationTable;


That comment "List of columns in a publication table" doesn't really
say anything helpful.

Perhaps it should mention that a NIL List means all table columns?

~~~

31. src/test/regress/sql/publication.sql

The regression test file has an uncommon mixture of /* */ and -- style comments.

Perhaps change all the /* */ ones?

~~~

32. src/test/regress/sql/publication.sql

+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);  -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);  -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);  -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);  -- ok

For all these tests (and more) there seems not sufficient explanation
comments to say exactly what each test case is testing, e.g. *why* is
an "error" expected for some cases but "ok" for others.

~~~

33. src/test/regress/sql/publication.sql

"-- no dice"

(??) confusing comment.

~~~

34. src/test/subscription/t/028_column_list.pl

I think a few more comments in this TAP file would help to make the
purpose of the tests more clear.

------
[1] https://www.postgresql.org/message-id/flat/CAHut%2BPtNWXPba0h%3Ddo_UiwaEziePNr7Z%2B58%2B-ctpyP2Pq1VkPw%40mail.gmail.com#76afd191811cba236198f62e60f44ade

Kind Regards,
Peter Smith.
Fujitsu Australia

  138. Re: Column Filtering in Logical Replication

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-01-31T07:13:47Z

    On 12.01.22 01:41, Alvaro Herrera wrote:
> I discovered a big hole in this, which is that ALTER PUBLICATION SET
> (publish='insert,update') can add UPDATE publishing to a publication
> that was only publishing INSERTs.  It's easy to implement a fix: in
> AlterPublicationOptions, scan the list of tables and raise an error if
> any of them has a column list that doesn't include all the columns in
> the replica identity.

Right now, we are not checking the publication options and the replica 
identity combinations at all at DDL time.  This is only checked at 
execution time in CheckCmdReplicaIdentity().  So under that scheme I 
don't think the check you describe is actually necessary.  Let the user 
set whatever combination they want, and check at execution time if it's 
an UPDATE or DELETE command whether the replica identity is sufficient.

  139. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-02-15T23:33:37Z

    Hi,

Here's an updated version of the patch, rebased to current master. Parts 
0002 and 0003 include various improvements based on review by me and 
another one by Peter Smith [1].

Part 0003 reworks and significantly extends the TAP test, to exercise 
various cases related to changes of replica identity etc. discussed in 
this thread. Some of the tests however still fail, because the behavior 
was not updated - I'll work on that once we agree what the expected 
behavior is.

1) partitioning with pubviaroot=true

The main set of failures is related to partitions with different replica 
identities and (pubviaroot=true), some of which may be mismatching the 
column list. There are multiple such test cases, depending on how the 
inconsistency is introduced - it may be there from the beginning, the 
column filter may be modified after adding the partitioned table to the 
publication, etc.

I think the expected behavior is to prohibit such cases from happening, 
by cross-checking the column filter when adding the partitioned table to 
publication, attaching a partition or changing a column filter.


2) merging multiple column filters

When the table has multiple column filters (in different publications), 
we need to merge them. Which works, except that FOR ALL TABLES [IN 
SCHEMA] needs to be handled as "has no column filter" (and replicates 
everything).


3) partitioning with pubivaroot=false

When a partitioned table is added with (pubviaroot=false), it should not 
be subject to column filter on the parent relation, which is the same 
behavior used by the row filtering patch.


regards


[1] 
https://www.postgresql.org/message-id/CAHut%2BPtc7Rh187eQKrxdUmUNWyfxz7OkhYAX%3DAW411Qwxya0LQ%40mail.gmail.com

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  140. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-02-15T23:57:31Z

    Hi Peter,

Thanks for the review and sorry for taking so long.

I've addressed most of the comments in the patch I sent a couple minutes 
ago. More comments in-line:


On 1/28/22 09:39, Peter Smith wrote:
> Here are some review comments for the v17-0001 patch.
> 
> ~~~
> 
> 1. Commit message
> 
> If no column list is specified, all the columns are replicated, as
> previously
> 
> Missing period (.) at the end of that sentence.
> 

I plan to reword that anyway.

> ~~~
> 
> 2. doc/src/sgml/catalogs.sgml
> 
> +      <para>
> +       This is an array of values that indicates which table columns are
> +       part of the publication.  For example a value of <literal>1 3</literal>
> +       would mean that the first and the third table columns are published.
> +       A null value indicates that all attributes are published.
> +      </para></entry>
> 
> Missing comma:
> "For example" --> "For example,"
> 

Fixed.

> Terms:
> The text seems to jump between "columns" and "attributes". Perhaps,
> for consistency, that last sentence should say: "A null value
> indicates that all columns are published."
> 

Yeah, but that's a pre-existing problem. I've modified the parts added 
by the patch to use "columns" though.

> ~~~
> 
> 3. doc/src/sgml/protocol.sgml
> 
>   </variablelist>
> -        Next, the following message part appears for each column
> (except generated columns):
> +        Next, the following message part appears for each column (except
> +        generated columns and other columns that don't appear in the column
> +        filter list, for tables that have one):
>   <variablelist>
> 
> Perhaps that can be expressed more simply, like:
> 
> Next, the following message part appears for each column (except
> generated columns and other columns not present in the optional column
> filter list):
> 

Not sure. I'll think about it.

> ~~~
> 
> 4. doc/src/sgml/ref/alter_publication.sgml
> 
> +ALTER PUBLICATION <replaceable class="parameter">name</replaceable>
> ALTER TABLE <replaceable
> class="parameter">publication_object</replaceable> SET COLUMNS { (
> <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
> 
> The syntax chart looks strange because there is already a "TABLE" and
> a column_name list within the "publication_object" definition, so do
> ALTER TABLE and publication_object co-exist?
> According to the current documentation it suggests nonsense like below is valid:
> ALTER PUBLICATION mypublication ALTER TABLE TABLE t1 (a,b,c) SET
> COLUMNS (a,b,c);
> 

Yeah, I think that's wrong. I think "publication_object" is wrong in 
this place, so I've used "table_name".

> --
> 
> But more fundamentally, I don't see why any new syntax is even needed at all.
> 
> Instead of:
> ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS
> (user_id, firstname, lastname);
> Why not just:
> ALTER PUBLICATION mypublication ALTER TABLE users (user_id, firstname,
> lastname);
> 

I haven't modified the grammar yet, but I agree SET COLUMNS seems a bit 
unnecessary. It also seems a bit inconsistent with ADD TABLE which 
simply lists the columns right adter the table name.

> Then, if the altered table defines a *different* column list then it
> would be functionally equivalent to whatever your SET COLUMNS is doing
> now. AFAIK this is how the Row-Filter [1] works, so that altering an
> existing table to have a different Row-Filter just overwrites that
> table's filter. IMO the Col-Filter behaviour should work the same as
> that - "SET COLUMNS" is redundant.
> 

I'm sorry, I don't understand what this is saying :-(

> ~~~
> 
> 5. doc/src/sgml/ref/alter_publication.sgml
> 
> -    TABLE [ ONLY ] <replaceable
> class="parameter">table_name</replaceable> [ * ] [, ... ]
> +    TABLE [ ONLY ] <replaceable
> class="parameter">table_name</replaceable> [ * ] [ ( <replaceable
> class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
> 
> That extra comma after the "column_name" seems wrong because there is
> one already in "[, ... ]".
> 

Fixed.

> ~~~
> 
> 6. doc/src/sgml/ref/create_publication.sgml
> 
> -    TABLE [ ONLY ] <replaceable
> class="parameter">table_name</replaceable> [ * ] [, ... ]
> +    TABLE [ ONLY ] <replaceable
> class="parameter">table_name</replaceable> [ * ] [ ( <replaceable
> class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
> 
> (Same as comment #5).
> That extra comma after the "column_name" seems wrong because there is
> one already in "[, ... ]".
> 

Fixed.

> ~~~
> 
> 7. doc/src/sgml/ref/create_publication.sgml
> 
> +     <para>
> +      When a column list is specified, only the listed columns are replicated;
> +      any other columns are ignored for the purpose of replication through
> +      this publication.  If no column list is specified, all columns of the
> +      table are replicated through this publication, including any columns
> +      added later.  If a column list is specified, it must include the replica
> +      identity columns.
> +     </para>
> 
> Suggest to re-word this a bit simpler:
> 
> e.g.
> - "listed columns" --> "named columns"
> - I don't think it is necessary to say the unlisted columns are ignored.
> - I didn't think it is necessary to say "though this publication"
> 
> AFTER
> When a column list is specified, only the named columns are replicated.
> If no column list is specified, all columns of the table are replicated,
> including any columns added later. If a column list is specified, it must
> include the replica identity columns.
> 

Fixed, seems reasonable.

> ~~~
> 
> 8. doc/src/sgml/ref/create_publication.sgml
> 
> Consider adding another example showing a CREATE PUBLICATION which has
> a column list.
> 

Added.

> ~~~
> 
> 9. src/backend/catalog/pg_publication.c - check_publication_add_relation
> 
>   /*
> - * Check if relation can be in given publication and throws appropriate
> - * error if not.
> + * Check if relation can be in given publication and that the column
> + * filter is sensible, and throws appropriate error if not.
> + *
> + * targetcols is the bitmapset of attribute numbers given in the column list,
> + * or NULL if it was not specified.
>    */
> 
> Typo: "targetcols" --> "columns" ??
> 

Right, I noticed that too.

> ~~~
> 
> 10. src/backend/catalog/pg_publication.c - check_publication_add_relation
> 
> +
> + /* Make sure the column list checks out */
> + if (columns != NULL)
> + {
> 
> Perhaps "checks out" could be worded better.
> 

Right, I expanded that in my review.

> ~~~
> 
> 11. src/backend/catalog/pg_publication.c - check_publication_add_relation
> 
> + /* Make sure the column list checks out */
> + if (columns != NULL)
> + {
> + /*
> + * Even if the user listed all columns in the column list, we cannot
> + * allow a column list to be specified when REPLICA IDENTITY is FULL;
> + * that would cause problems if a new column is added later, because
> + * the new column would have to be included (because of being part of
> + * the replica identity) but it's technically not allowed (because of
> + * not being in the publication's column list yet).  So reject this
> + * case altogether.
> + */
> + if (replidentfull)
> + ereport(ERROR,
> + errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> + errmsg("invalid column list for publishing relation \"%s\"",
> +    RelationGetRelationName(targetrel)),
> + errdetail("Cannot specify a column list on relations with REPLICA
> IDENTITY FULL."));
> +
> + check_publication_columns(pub, targetrel, columns);
> + }
> 
> IIUC almost all of the above comment and code is redundant because by
> calling the check_publication_columns function it will do exactly the
> same check...
> 
> So,  that entire slab might be replaced by 2 lines:
> 
> if (columns != NULL)
> check_publication_columns(pub, targetrel, columns);
> 

You're right. But I think we can make that even simpler by moving even 
the (columns!=NULL) check into the function.

> ~~~
> 
> 12. src/backend/catalog/pg_publication.c - publication_set_table_columns
> 
> +publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
> +   Relation targetrel, List *columns)
> +{
> + Bitmapset  *attset;
> + AttrNumber *attarray;
> + HeapTuple copytup;
> + int natts;
> + bool nulls[Natts_pg_publication_rel];
> + bool replaces[Natts_pg_publication_rel];
> + Datum values[Natts_pg_publication_rel];
> +
> + memset(values, 0, sizeof(values));
> + memset(nulls, 0, sizeof(nulls));
> + memset(replaces, false, sizeof(replaces));
> 
> It seemed curious to use memset false for "replaces" but memset 0 for
> "nulls", since they are both bool arrays (??)
> 

Fixed.

> ~~~
> 
> 13. src/backend/catalog/pg_publication.c - compare_int16
> 
> +/* qsort comparator for attnums */
> +static int
> +compare_int16(const void *a, const void *b)
> +{
> + int av = *(const int16 *) a;
> + int bv = *(const int16 *) b;
> +
> + /* this can't overflow if int is wider than int16 */
> + return (av - bv);
> +}
> 
> This comparator seems common with another one already in the PG
> source. Perhaps it would be better for generic comparators (like this
> one) to be in some common code instead of scattered cut/paste copies
> of the same thing.
> 

I thought about it, but it doesn't really seem worth the effort.

> ~~~
> 
> 14. src/backend/commands/publicationcmds.c - AlterPublicationTables
> 
> + else if (stmt->action == AP_SetColumns)
> + {
> + Assert(schemaidlist == NIL);
> + Assert(list_length(tables) == 1);
> +
> + PublicationSetColumns(stmt, pubform,
> +   linitial_node(PublicationTable, tables));
> + }
> 
> (Same as my earlier review comment #4)
> 
> Suggest to call this PublicationSetColumns based on some smarter
> detection logic of a changed column list. Please refer to the
> Row-Filter patch [1] for this same function.
> 

I don't understand. Comment #4 is about syntax, no?

> ~~~
> 
> 15. src/backend/commands/publicationcmds.c - AlterPublicationTables
> 
> + /* This is not needed to delete a table */
> + pubrel->columns = NIL;
> 
> Perhaps a more explanatory comment would be better there?
> 

If I understand the comment, it says we don't actually need to set 
columns to NIL. In which case we can just get rid of the change.

> ~~~
> 
> 16. src/backend/commands/tablecmds.c - relation_mark_replica_identity
> 
> @@ -15841,6 +15871,7 @@ relation_mark_replica_identity(Relation rel,
> char ri_type, Oid indexOid,
>    CatalogTupleUpdate(pg_index, &pg_index_tuple->t_self, pg_index_tuple);
>    InvokeObjectPostAlterHookArg(IndexRelationId, thisIndexOid, 0,
>    InvalidOid, is_internal);
> +
>    /*
>    * Invalidate the relcache for the table, so that after we commit
>    * all sessions will refresh the table's replica identity index
> 
> Spurious whitespace change seemed unrelated to the Col-Filter patch.
> 

Fixed.

> ~~~
> 
> 17. src/backend/parser/gram.y
> 
>    *
> + * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
> + * ALTER PUBLICATION name SET COLUMNS table_name ALL
> + *
> 
> (Same as my earlier review comment #4)
> 
> IMO there was no need for the new syntax of SET COLUMNS.
> 

Not modified yet, we'll see about the syntax.

> ~~~
> 
> 18. src/backend/replication/logical/proto.c - logicalrep_write_attrs
> 
> - /* send number of live attributes */
> - for (i = 0; i < desc->natts; i++)
> - {
> - if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc,
> i)->attgenerated)
> - continue;
> - nliveatts++;
> - }
> - pq_sendint16(out, nliveatts);
> -
>    /* fetch bitmap of REPLICATION IDENTITY attributes */
>    replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
>    if (!replidentfull)
>    idattrs = RelationGetIdentityKeyBitmap(rel);
> 
> + /* send number of live attributes */
> + for (i = 0; i < desc->natts; i++)
> + {
> + Form_pg_attribute att = TupleDescAttr(desc, i);
> +
> + if (att->attisdropped || att->attgenerated)
> + continue;
> + if (columns != NULL && !bms_is_member(att->attnum, columns))
> + continue;
> + nliveatts++;
> + }
> + pq_sendint16(out, nliveatts);
> +
> 
> This change seemed to have the effect of moving that 4 lines of
> "replidentfull" code from below the loop to above the loop. But moving
> that code seems unrelated to the Col-Filter patch. (??).
> 

Right, restored the original code.

> ~~~
> 
> 19. src/backend/replication/logical/tablesync.c - fetch_remote_table_info
> 
> @@ -793,12 +877,12 @@ fetch_remote_table_info(char *nspname, char *relname,
> 
>    ExecClearTuple(slot);
>    }
> +
>    ExecDropSingleTupleTableSlot(slot);
> -
> - lrel->natts = natt;
> -
>    walrcv_clear_result(res);
>    pfree(cmd.data);
> +
> + lrel->natts = natt;
>   }
> 
> The shuffling of those few lines seems unrelated to any requirement of
> the Col-Filter patch (??)
> 

Yep, undone. I'd bet this is simply due to older versions of the patch 
touching this place, and then undoing some of it.

> ~~~
> 
> 20. src/backend/replication/logical/tablesync.c - copy_table
> 
> + for (int i = 0; i < lrel.natts; i++)
> + {
> + appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
> + if (i < lrel.natts - 1)
> + appendStringInfoString(&cmd, ", ");
> + }
> 
> Perhaps that could be expressed more simply if the other way around like:
> 
> for (int i = 0; i < lrel.natts; i++)
> {
> if (i)
> appendStringInfoString(&cmd, ", ");
> appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
> }
> 

I used a slightly different version.

> ~~~
> 
> 21. src/backend/replication/pgoutput/pgoutput.c
> 
> +
> + /*
> + * Set of columns included in the publication, or NULL if all columns are
> + * included implicitly.  Note that the attnums in this list are not
> + * shifted by FirstLowInvalidHeapAttributeNumber.
> + */
> + Bitmapset  *columns;
> 
> Typo: "in this list" --> "in this set" (??)
> 

"bitmap" is what we call Bitmapset so I used that.

> ~~~
> 
> 22. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
> 
>    * Don't publish changes for partitioned tables, because
> - * publishing those of its partitions suffices, unless partition
> - * changes won't be published due to pubviaroot being set.
> + * publishing those of its partitions suffices.  (However, ignore
> + * this if partition changes are not to published due to
> + * pubviaroot being set.)
>    */
> 
> This change seems unrelated to the Col-Filter patch, so perhaps it
> should not be here at all.
> 
> Also, typo: "are not to published"
> 

Yeah, unrelated. Reverted.

> ~~~
> 
> 23. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
> 
> + /*
> + * Obtain columns published by this publication, and add them
> + * to the list for this rel.  Note that if at least one
> + * publication has a empty column list, that means to publish
> + * everything; so if we saw a publication that includes all
> + * columns, skip this.
> + */
> 
> Typo: "a empty" --> "an empty"
> 

Fixed.

> ~~~
> 
> 24. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
> 
> + if (isnull)
> + {
> + /*
> + * If we see a publication with no columns, reset the
> + * list and ignore further ones.
> + */
> 
> Perhaps that comment is meant to say "with no column filter" instead
> of "with no columns"?
> 

Yep, fixed.

> ~~~
> 
> 25. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
> 
> + if (isnull)
> + {
> ...
> + }
> + else if (!isnull)
> + {
> ...
> + }
> 
> Is the "if (!isnull)" in the else just to be really REALLY sure it is not null?
> 

Double-tap ;-) Removed the condition.

> ~~~
> 
> 26. src/bin/pg_dump/pg_dump.c - getPublicationTables
> 
> + pubrinfo[i].pubrattrs = attribs->data;
> + }
> + else
> + pubrinfo[j].pubrattrs = NULL;
> 
> I got confused reading this code. Are those different indices 'i' and
> 'j' correct?
> 

Good catch! I think you're right and it should be "j" in both places. 
This'd only cause trouble in selective pg_dumps (when dumping selected 
tables). The patch clearly needs some pg_dump tests.

> ~~~
> 
> 27. src/bin/psql/describe.c
> 
> The Row-Filter [1] displays filter information not only for the psql
> \dRp+ command but also for the psql \d <tablename> command. Perhaps
> the Col-Filter patch should do that too.
> 

Not sure.

> ~~~
> 
> 28. src/bin/psql/tab-complete.c
> 
> @@ -1657,6 +1657,8 @@ psql_completion(const char *text, int start, int end)
>    /* ALTER PUBLICATION <name> ADD */
>    else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
>    COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
> + else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
> + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
>    /* ALTER PUBLICATION <name> DROP */
> 
> I am not sure about this one- is that change even related to the
> Col-Filter patch or is this some unrelated bugfix?
> 

Yeah, seems unrelated - possibly from a rebase or something. Removed.

> ~~~
> 
> 29. src/include/catalog/pg_publication.h
> 
> @@ -86,6 +86,7 @@ typedef struct Publication
>   typedef struct PublicationRelInfo
>   {
>    Relation relation;
> + List    *columns;
>   } PublicationRelInfo;
> 
> Perhaps that needs some comment. e.g. do you need to mention that a
> NIL List means all columns?
> 

I added a short comment.

> ~~~
> 
> 30. src/include/nodes/parsenodes.h
> 
> @@ -3642,6 +3642,7 @@ typedef struct PublicationTable
>   {
>    NodeTag type;
>    RangeVar   *relation; /* relation to be published */
> + List    *columns; /* List of columns in a publication table */
>   } PublicationTable;
> 
> 
> That comment "List of columns in a publication table" doesn't really
> say anything helpful.
> 
> Perhaps it should mention that a NIL List means all table columns?
> 

Not sure, seems fine.

> ~~~
> 
> 31. src/test/regress/sql/publication.sql
> 
> The regression test file has an uncommon mixture of /* */ and -- style comments.
> 
> Perhaps change all the /* */ ones?
> 

Yeah, that needs some cleanup. I haven't done anything about it yet.

> ~~~
> 
> 32. src/test/regress/sql/publication.sql
> 
> +CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
> + d int generated always as (a + length(b)) stored);
> +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);  -- error
> +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);  -- error
> +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);  -- error
> +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);  -- ok
> 
> For all these tests (and more) there seems not sufficient explanation
> comments to say exactly what each test case is testing, e.g. *why* is
> an "error" expected for some cases but "ok" for others.
> 

Not sure. I think the error is generally obvious in the expected output.

> ~~~
> 
> 33. src/test/regress/sql/publication.sql
> 
> "-- no dice"
> 
> (??) confusing comment.
> 

Same as for the errors.

> ~~~
> 
> 34. src/test/subscription/t/028_column_list.pl
> 
> I think a few more comments in this TAP file would help to make the
> purpose of the tests more clear.
> 

Yeah, the 0004 patch I shared a couple minutes ago does exactly that.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  141. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-02-16T00:33:32Z

    On 2022-Feb-16, Tomas Vondra wrote:

> Here's an updated version of the patch, rebased to current master. Parts
> 0002 and 0003 include various improvements based on review by me and another
> one by Peter Smith [1].

Thanks for doing this!

> 1) partitioning with pubviaroot=true

I agree that preventing the inconsistencies from happening is probably
the best.

> 2) merging multiple column filters
> 
> When the table has multiple column filters (in different publications), we
> need to merge them. Which works, except that FOR ALL TABLES [IN SCHEMA]
> needs to be handled as "has no column filter" (and replicates everything).

Agreed.

> 3) partitioning with pubivaroot=false
> 
> When a partitioned table is added with (pubviaroot=false), it should not be
> subject to column filter on the parent relation, which is the same behavior
> used by the row filtering patch.

You mean each partition should define its own filter, or lack of filter?
That sounds reasonable.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"Pensar que el espectro que vemos es ilusorio no lo despoja de espanto,
sólo le suma el nuevo terror de la locura" (Perelandra, C.S. Lewis)

  142. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-02-16T00:39:37Z

    
On 2/16/22 01:33, Alvaro Herrera wrote:
> On 2022-Feb-16, Tomas Vondra wrote:
> 
>> Here's an updated version of the patch, rebased to current master. Parts
>> 0002 and 0003 include various improvements based on review by me and another
>> one by Peter Smith [1].
> 
> Thanks for doing this!
> 
>> 1) partitioning with pubviaroot=true
> 
> I agree that preventing the inconsistencies from happening is probably
> the best.
> 
>> 2) merging multiple column filters
>>
>> When the table has multiple column filters (in different publications), we
>> need to merge them. Which works, except that FOR ALL TABLES [IN SCHEMA]
>> needs to be handled as "has no column filter" (and replicates everything).
> 
> Agreed.
> 
>> 3) partitioning with pubivaroot=false
>>
>> When a partitioned table is added with (pubviaroot=false), it should not be
>> subject to column filter on the parent relation, which is the same behavior
>> used by the row filtering patch.
> 
> You mean each partition should define its own filter, or lack of filter?
> That sounds reasonable.
> 

If the partition is not published by the root, it shouldn't use the
filter defined on the root. I wonder what should happen to the filter
defined on the partition itself. I'd say

pubviaroot=false -> use filter defined on partition (if any)

pubviaroot=true -> use filter defined on root (if any)


I wonder what the row filter patch is doing - we should probably follow
the same logic, if only to keep the filtering stuff consistent.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  143. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-02-16T09:30:29Z

    On Wed, Feb 16, 2022 at 6:09 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 2/16/22 01:33, Alvaro Herrera wrote:
> >
> >> 3) partitioning with pubivaroot=false
> >>
> >> When a partitioned table is added with (pubviaroot=false), it should not be
> >> subject to column filter on the parent relation, which is the same behavior
> >> used by the row filtering patch.
> >
> > You mean each partition should define its own filter, or lack of filter?
> > That sounds reasonable.
> >
>
> If the partition is not published by the root, it shouldn't use the
> filter defined on the root. I wonder what should happen to the filter
> defined on the partition itself. I'd say
>
> pubviaroot=false -> use filter defined on partition (if any)
>
> pubviaroot=true -> use filter defined on root (if any)
>
>
> I wonder what the row filter patch is doing - we should probably follow
> the same logic, if only to keep the filtering stuff consistent.
>

The row filter patch is doing the same and additionally, it gives an
error if the user provides a filter for a partitioned table with
pubviaroot as false.

-- 
With Regards,
Amit Kapila.

  144. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-02-16T10:05:34Z

    On Wed, Feb 16, 2022 at 5:03 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> Hi,
>
> Here's an updated version of the patch, rebased to current master. Parts
> 0002 and 0003 include various improvements based on review by me and
> another one by Peter Smith [1].
>
> Part 0003 reworks and significantly extends the TAP test, to exercise
> various cases related to changes of replica identity etc. discussed in
> this thread. Some of the tests however still fail, because the behavior
> was not updated - I'll work on that once we agree what the expected
> behavior is.
>
> 1) partitioning with pubviaroot=true
>
> The main set of failures is related to partitions with different replica
> identities and (pubviaroot=true), some of which may be mismatching the
> column list. There are multiple such test cases, depending on how the
> inconsistency is introduced - it may be there from the beginning, the
> column filter may be modified after adding the partitioned table to the
> publication, etc.
>
> I think the expected behavior is to prohibit such cases from happening,
> by cross-checking the column filter when adding the partitioned table to
> publication, attaching a partition or changing a column filter.
>

I feel it is better to follow the way described by Peter E. here [1]
to handle these cases. The row filter patch is also using the same
scheme as that is what we are doing now for Updates/Deletes and it
would be really challenging and much more effort/code to deal with
everything at DDL time. I have tried to explain some of that in my
emails [2][3].

[1] - https://www.postgresql.org/message-id/ca91dc91-80ba-e954-213e-b4170a6160f5%40enterprisedb.com
[2] - https://www.postgresql.org/message-id/CAA4eK1%2Bm45Xyzx7AUY9TyFnB6CZ7_%2B_uooPb7WHSpp7UE%3DYmKg%40mail.gmail.com
[3] - https://www.postgresql.org/message-id/CAA4eK1%2B1DMkCip9SB3B0_u0Q6fGf-D3vgqQodkLfur0qkL482g%40mail.gmail.com


-- 
With Regards,
Amit Kapila.

  145. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-02T12:13:15Z

    Hi,

Attached is an updated patch, addressing most of the issues reported so
far. There are various minor tweaks, but the main changes are:

1) regular regression tests, verifying (hopefully) all the various cases
of publication vs. column filters, replica identity check at various
changes and so on

2) pg_dump tests, testing column filters (alone and with row filter)

3) checks of column filter vs. publish_via_partition_root and replica
identity, following the same logic as the row-filter patch (hopefully,
it touches the same places, using the same logic, ...)

That means - with "publish_via_partition_root=false" it's not allowed to
specify column filters on partitioned tables, only for leaf partitions.

And we check column filter vs. replica identity when adding tables to
publications, or whenever we change the replica identity.


The patch is still a bit crude, I'm sure some of the places (especially
the new ones) may need cleanup/recovery. But I think it's much closer to
being committable, I think.

The first two simple patches are adding tests for the row filtering. So
this is not really part of this patch.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  146. Re: Column Filtering in Logical Replication

    Justin Pryzby <pryzby@telsasoft.com> — 2022-03-03T00:29:21Z

    I applied this patch in my branch with CI hacks to show code coverage on
cirrus.
https://api.cirrus-ci.com/v1/artifact/task/6186186539532288/coverage/coverage/00-index.html

Eyeballing it looks good.  But GetActionsInPublication() isn't being hit at
all?

I think the queries in pg_dump should be written with the common portions of
the query outside the conditional.

-- 
Justin

  147. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-04T10:42:30Z

    On Wed, Mar 2, 2022 at 5:43 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> Attached is an updated patch, addressing most of the issues reported so
> far. There are various minor tweaks, but the main changes are:
...
>
> 3) checks of column filter vs. publish_via_partition_root and replica
> identity, following the same logic as the row-filter patch (hopefully,
> it touches the same places, using the same logic, ...)
>
> That means - with "publish_via_partition_root=false" it's not allowed to
> specify column filters on partitioned tables, only for leaf partitions.
>
> And we check column filter vs. replica identity when adding tables to
> publications, or whenever we change the replica identity.
>

This handling is different from row filter work and I see problems
with it. The column list validation w.r.t primary key (default replica
identity) is missing. The handling of column list vs. partitions has
multiple problems: (a) In attach partition, the patch is just checking
ancestors for RI validation but what if the table being attached has
further subpartitions; (b) I think the current locking also seems to
have problems because it is quite possible that while it validates the
ancestors here, concurrently someone changes the column list. I think
it won't be enough to just change the locking mode because with the
current patch strategy during attach, we will be first taking locks
for child tables of current partition and then parent tables which can
pose deadlock hazards.

The columns list validation also needs to be done when we change
publication action.

There could be more similar problems which I might have missed. For
some of these (except for concurrency issues), my colleague Shi-San
has done testing and the results are below [1]. I feel we should do RI
vs. column list handling similar to row filter work (at one place) to
avoid all such hazards and possibly similar handling at various
places, there is a good chance that we will miss some places or make
mistakes that are not easy to catch. Do let me know if you think it
makes sense for me or one of the people who work on row filter patch
to try this (make the handling of RI checks similar to row filter
work) and then we can see if that turns out to be a simple way to deal
with all these problems?

Some other miscellaneous comments:
=============================
*
In get_rel_sync_entry(), the handling for partitioned tables doesn't
seem to be correct. It can publish a different set of columns based on
the order of publications specified in the subscription.

For example:
----
create table parent (a int, b int, c int) partition by range (a);
create table test_part1 (like parent);
alter table parent attach partition test_part1 for values from (1) to (10);

create publication pub for table parent(a) with (PUBLISH_VIA_PARTITION_ROOT);
create publication pub2 for table test_part1(b);
---

Now, depending on the order of publications in the list while defining
subscription, the column list will change
----
create subscription sub connection 'port=10000 dbname=postgres'
publication pub, pub2;

For the above, column list will be: (a)

create subscription sub connection 'port=10000 dbname=postgres'
publication pub2, pub;

For this one, the column list will be: (a, b)
----

To avoid this, the column list should be computed based on the final
publish_as_relid as we are doing for the row filter.

*
Fetching column filter info in tablesync.c is quite expensive. It
seems to be using four round-trips to get the complete info whereas
for row-filter we use just one round trip. I think we should try to
get both row filter and column filter info in just one round trip.

[1] -
Test-1:
The patch doesn't check when the primary key changes.

e.g.
-- publisher --
create table tbl(a int primary key, b int);
create publication pub for table tbl(a);
alter table tbl drop CONSTRAINT tbl_pkey;
alter table tbl add primary key (b);
insert into tbl values (1,1);

-- subscriber --
create table tbl(a int, b int);
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
update tbl set b=1 where a=1;
alter table tbl add primary key (b);

-- publisher --
delete from tbl;

Column "b" is part of replica identity, but it is filtered, which
caused an error on the subscriber side.

ERROR:  publisher did not send replica identity column expected by the
logical replication target relation "public.tbl"
CONTEXT:  processing remote data during "DELETE" for replication
target relation "public.tbl" in transaction 724 at 2022-03-04
11:46:16.330892+08

Test-2: Partitioned table RI w.r.t column list.
2.1
Using "create table ... partition of".

e.g.
-- publisher --
create table parent (a int, b int) partition by range (a);
create publication pub for table parent(a)
with(publish_via_partition_root=true);
create table child partition of parent (primary key (a,b)) default;
insert into parent values (1,1);

-- subscriber --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres'
publication pub; update child set b=1 where a=1;
alter table parent add primary key (a,b);

-- publisher --
delete from parent;

Column "b" is part of replica identity in the child table, but it is
filtered, which caused an error on the subscriber side.

ERROR:  publisher did not send replica identity column expected by the
logical replication target relation "public.parent"
CONTEXT:  processing remote data during "DELETE" for replication
target relation "public.parent" in transaction 723 at 2022-03-04
15:15:39.776949+08

2.2
It is likely that a table to be attached also has a partition.

e.g.
-- publisher --
create table t1 (a int, b int) partition by range (a);
create publication pub for table t1(b) with(publish_via_partition_root=true);
create table t2 (a int, b int) partition by range (a);
create table t3 (a int primary key, b int);
alter table t2 attach partition t3 default;
alter table t1 attach partition t2 default;
insert into t1 values (1,1);

-- subscriber --
create table t1 (a int, b int) partition by range (a);
create table t2 (a int, b int) partition by range (a);
create table t3 (a int, b int);
alter table t2 attach partition t3 default;
alter table t1 attach partition t2 default;
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
update t1 set a=1 where b=1;
alter table t1 add primary key (a);

-- publisher --
delete from t1;

Column "a" is part of replica identity in table t3, but t3's ancestor
t1 is published with column "a" filtered, which caused an error on the
subscriber side.

ERROR:  publisher did not send replica identity column expected by the
logical replication target relation "public.t1"
CONTEXT:  processing remote data during "DELETE" for replication
target relation "public.t1" in transaction 726 at 2022-03-04
14:40:29.297392+08

3.
Using "alter publication pub set(publish='...'); "

e.g.
-- publisher --
create table tbl(a int primary key, b int); create publication pub for
table tbl(b) with(publish='insert'); insert into tbl values (1,1);

-- subscriber --
create table tbl(a int, b int);
create subscription sub connection 'port=5432 dbname=postgres' publication pub;

-- publisher --
alter publication pub set(publish='insert,update');

-- subscriber --
update tbl set a=1 where b=1;
alter table tbl add primary key (b);

-- publisher --
update tbl set a=2 where a=1;

Updates are replicated, and the column "a" is part of replica
identity, but it is filtered, which caused an error on the subscriber
side.

ERROR:  publisher did not send replica identity column expected by the
logical replication target relation "public.tbl"
CONTEXT:  processing remote data during "UPDATE" for replication
target relation "public.tbl" in transaction 723 at 2022-03-04
11:56:33.905843+08

-- 
With Regards,
Amit Kapila.

  148. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-07T15:18:04Z

    
On 3/4/22 11:42, Amit Kapila wrote:
> On Wed, Mar 2, 2022 at 5:43 PM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> Attached is an updated patch, addressing most of the issues reported so
>> far. There are various minor tweaks, but the main changes are:
> ...
>>
>> 3) checks of column filter vs. publish_via_partition_root and replica
>> identity, following the same logic as the row-filter patch (hopefully,
>> it touches the same places, using the same logic, ...)
>>
>> That means - with "publish_via_partition_root=false" it's not allowed to
>> specify column filters on partitioned tables, only for leaf partitions.
>>
>> And we check column filter vs. replica identity when adding tables to
>> publications, or whenever we change the replica identity.
>>
> 
> This handling is different from row filter work and I see problems
> with it.

By different, I assume you mean I tried to enfoce the rules in ALTER
PUBLICATION and other ALTER commands, instead of when modifying the
data? OK, I reworked this to do the same thing as the row filtering patch.

> The column list validation w.r.t primary key (default replica
> identity) is missing. The handling of column list vs. partitions has
> multiple problems: (a) In attach partition, the patch is just checking
> ancestors for RI validation but what if the table being attached has
> further subpartitions; (b) I think the current locking also seems to
> have problems because it is quite possible that while it validates the
> ancestors here, concurrently someone changes the column list. I think
> it won't be enough to just change the locking mode because with the
> current patch strategy during attach, we will be first taking locks
> for child tables of current partition and then parent tables which can
> pose deadlock hazards.
> > The columns list validation also needs to be done when we change
> publication action.
>
I believe those issues should be solved by adopting the same approach as
the row-filtering patch, right?

> There could be more similar problems which I might have missed. For
> some of these (except for concurrency issues), my colleague Shi-San
> has done testing and the results are below [1]. I feel we should do RI
> vs. column list handling similar to row filter work (at one place) to
> avoid all such hazards and possibly similar handling at various
> places, there is a good chance that we will miss some places or make
> mistakes that are not easy to catch.

I agree if both patches use the same approach, that would reduce the
risk of missing the handling in one place, etc.

> Do let me know if you think it makes sense for me or one of the 
> people who work on row filter patch to try this (make the handling of
> RI checks similar to row filter work) and then we can see if that
> turns out to be a simple way to deal with all these problems?
> 

If someone who is more familiar with the design conclusions from the row
filtering patch, that would be immensely useful. Especially now, when I
reworked it to the same approach as the row filtering patch.


> Some other miscellaneous comments:
> =============================
> *
> In get_rel_sync_entry(), the handling for partitioned tables doesn't
> seem to be correct. It can publish a different set of columns based on
> the order of publications specified in the subscription.
> 
> For example:
> ----
> create table parent (a int, b int, c int) partition by range (a);
> create table test_part1 (like parent);
> alter table parent attach partition test_part1 for values from (1) to (10);
> 
> create publication pub for table parent(a) with (PUBLISH_VIA_PARTITION_ROOT);
> create publication pub2 for table test_part1(b);
> ---
> 
> Now, depending on the order of publications in the list while defining
> subscription, the column list will change
> ----
> create subscription sub connection 'port=10000 dbname=postgres'
> publication pub, pub2;
> 
> For the above, column list will be: (a)
> 
> create subscription sub connection 'port=10000 dbname=postgres'
> publication pub2, pub;
> 
> For this one, the column list will be: (a, b)
> ----
> 
> To avoid this, the column list should be computed based on the final
> publish_as_relid as we are doing for the row filter.
> 

Hmm, yeah. That seems like a genuine problem - it should not depend on
the order of publications in the subscription, I guess.

But is it an issue in the patch? Isn't that a pre-existing issue? AFAICS
the problem is that we initialize publish_as_relid=relid before the loop
over publications, and then just update it. So the first iteration
starts with relid, but the second iteration ends with whatever value is
set by the first iteration (e.g. the root).

So with the example you posted, we start with

  publish_as_relid = relid = test_part1

but then if the first publication is pubviaroot=true, we update it to
parent. And in the second iteration, we fail to find the column filter,
because "parent" (publish_as_relid) is not part of the pub2.

If we do it in the other order, we leave the publish_as_relid value as
is (and find the filter), and then update it in the second iteration
(and find the column filter too).

Now, this can be resolved by re-calculating the publish_as_relid from
scratch in each iteration (start with relid, then maybe update it). But
that's just half the story - the issue is there even without column
filters. Consider this example:

create table t (a int, b int, c int) partition by range (a);

create table t_1 partition of t for values from (1) to (10)
       partition by range (a);

create table t_2 partition of t_1 for values from (1) to (10);

create publication pub1 for table t(a)
  with (PUBLISH_VIA_PARTITION_ROOT);

create publication pub2 for table t_1(a)
  with (PUBLISH_VIA_PARTITION_ROOT);


Now, is you change subscribe to "pub1, pub2" and "pub2, pub1", we'll end
up with different publish_as_relid values (t or t_1). Which seems like
the same ambiguity issue.


> *
> Fetching column filter info in tablesync.c is quite expensive. It
> seems to be using four round-trips to get the complete info whereas
> for row-filter we use just one round trip. I think we should try to
> get both row filter and column filter info in just one round trip.
> 

Maybe, but I really don't think this is an issue. The sync happens only
very rarely, and the rest of the sync (starting workers, copying data)
is likely way more expensive than this.

> [1] -
> Test-1:
> The patch doesn't check when the primary key changes.
> 
> e.g.
> -- publisher --
> create table tbl(a int primary key, b int);
> create publication pub for table tbl(a);
> alter table tbl drop CONSTRAINT tbl_pkey;
> alter table tbl add primary key (b);
> insert into tbl values (1,1);
> 
> -- subscriber --
> create table tbl(a int, b int);
> create subscription sub connection 'port=5432 dbname=postgres' publication pub;
> update tbl set b=1 where a=1;
> alter table tbl add primary key (b);
> 
> -- publisher --
> delete from tbl;
> 
> Column "b" is part of replica identity, but it is filtered, which
> caused an error on the subscriber side.
> 
> ERROR:  publisher did not send replica identity column expected by the
> logical replication target relation "public.tbl"
> CONTEXT:  processing remote data during "DELETE" for replication
> target relation "public.tbl" in transaction 724 at 2022-03-04
> 11:46:16.330892+08
> 
> Test-2: Partitioned table RI w.r.t column list.
> 2.1
> Using "create table ... partition of".
> 
> e.g.
> -- publisher --
> create table parent (a int, b int) partition by range (a);
> create publication pub for table parent(a)
> with(publish_via_partition_root=true);
> create table child partition of parent (primary key (a,b)) default;
> insert into parent values (1,1);
> 
> -- subscriber --
> create table parent (a int, b int) partition by range (a);
> create table child partition of parent default;
> create subscription sub connection 'port=5432 dbname=postgres'
> publication pub; update child set b=1 where a=1;
> alter table parent add primary key (a,b);
> 
> -- publisher --
> delete from parent;
> 
> Column "b" is part of replica identity in the child table, but it is
> filtered, which caused an error on the subscriber side.
> 
> ERROR:  publisher did not send replica identity column expected by the
> logical replication target relation "public.parent"
> CONTEXT:  processing remote data during "DELETE" for replication
> target relation "public.parent" in transaction 723 at 2022-03-04
> 15:15:39.776949+08
> 
> 2.2
> It is likely that a table to be attached also has a partition.
> 
> e.g.
> -- publisher --
> create table t1 (a int, b int) partition by range (a);
> create publication pub for table t1(b) with(publish_via_partition_root=true);
> create table t2 (a int, b int) partition by range (a);
> create table t3 (a int primary key, b int);
> alter table t2 attach partition t3 default;
> alter table t1 attach partition t2 default;
> insert into t1 values (1,1);
> 
> -- subscriber --
> create table t1 (a int, b int) partition by range (a);
> create table t2 (a int, b int) partition by range (a);
> create table t3 (a int, b int);
> alter table t2 attach partition t3 default;
> alter table t1 attach partition t2 default;
> create subscription sub connection 'port=5432 dbname=postgres' publication pub;
> update t1 set a=1 where b=1;
> alter table t1 add primary key (a);
> 
> -- publisher --
> delete from t1;
> 
> Column "a" is part of replica identity in table t3, but t3's ancestor
> t1 is published with column "a" filtered, which caused an error on the
> subscriber side.
> 
> ERROR:  publisher did not send replica identity column expected by the
> logical replication target relation "public.t1"
> CONTEXT:  processing remote data during "DELETE" for replication
> target relation "public.t1" in transaction 726 at 2022-03-04
> 14:40:29.297392+08
> 
> 3.
> Using "alter publication pub set(publish='...'); "
> 
> e.g.
> -- publisher --
> create table tbl(a int primary key, b int); create publication pub for
> table tbl(b) with(publish='insert'); insert into tbl values (1,1);
> 
> -- subscriber --
> create table tbl(a int, b int);
> create subscription sub connection 'port=5432 dbname=postgres' publication pub;
> 
> -- publisher --
> alter publication pub set(publish='insert,update');
> 
> -- subscriber --
> update tbl set a=1 where b=1;
> alter table tbl add primary key (b);
> 
> -- publisher --
> update tbl set a=2 where a=1;
> 
> Updates are replicated, and the column "a" is part of replica
> identity, but it is filtered, which caused an error on the subscriber
> side.
> 
> ERROR:  publisher did not send replica identity column expected by the
> logical replication target relation "public.tbl"
> CONTEXT:  processing remote data during "UPDATE" for replication
> target relation "public.tbl" in transaction 723 at 2022-03-04
> 11:56:33.905843+08
> 

AFAICS these issues should be resolved by the adoption of the row-filter
approach (i.e. it should fail the same way as for row filter).


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  149. Re: Column Filtering in Logical Replication

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-03-09T09:20:57Z

    On 07.03.22 16:18, Tomas Vondra wrote:
> AFAICS these issues should be resolved by the adoption of the row-filter
> approach (i.e. it should fail the same way as for row filter).

The first two patches (additional testing for row filtering feature) 
look okay to me.

Attached is a fixup patch for your main feature patch (the third one).

It's a bit of code and documentation cleanup, but mainly I removed the 
term "column filter" from the patch.  Half the code was using "column 
list" or similar and half the code "column filter", which was confusing. 
  Also, there seemed to be a bit of copy-and-pasting from row-filter 
code going on, with some code comments not quite sensible, so I rewrote 
some of them.  Also some code used "rf" and "cf" symbols which were a 
bit hard to tell apart.  A few more letters can increase readability.

Note in publicationcmds.c OpenTableList() the wrong if condition was used.

I'm still confused about the intended replica identity handling.  This 
patch still checks whether the column list contains the replica identity 
at DDL time.  And then it also checks at execution time.  I thought the 
latest understanding was that the DDL-time checking would be removed.  I 
think it's basically useless now, since as the test cases show, you can 
subvert those checks by altering the replica identity later.

  150. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-09T10:03:40Z

    On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/4/22 11:42, Amit Kapila wrote:
> > On Wed, Mar 2, 2022 at 5:43 PM Tomas Vondra
> > <tomas.vondra@enterprisedb.com> wrote:
> >>
> >> Attached is an updated patch, addressing most of the issues reported so
> >> far. There are various minor tweaks, but the main changes are:
> > ...
> >>
> >> 3) checks of column filter vs. publish_via_partition_root and replica
> >> identity, following the same logic as the row-filter patch (hopefully,
> >> it touches the same places, using the same logic, ...)
> >>
> >> That means - with "publish_via_partition_root=false" it's not allowed to
> >> specify column filters on partitioned tables, only for leaf partitions.
> >>
> >> And we check column filter vs. replica identity when adding tables to
> >> publications, or whenever we change the replica identity.
> >>
> >
> > This handling is different from row filter work and I see problems
> > with it.
>
> By different, I assume you mean I tried to enfoce the rules in ALTER
> PUBLICATION and other ALTER commands, instead of when modifying the
> data?
>

Yes.

> OK, I reworked this to do the same thing as the row filtering patch.
>

Thanks, I'll check this.

> > The column list validation w.r.t primary key (default replica
> > identity) is missing. The handling of column list vs. partitions has
> > multiple problems: (a) In attach partition, the patch is just checking
> > ancestors for RI validation but what if the table being attached has
> > further subpartitions; (b) I think the current locking also seems to
> > have problems because it is quite possible that while it validates the
> > ancestors here, concurrently someone changes the column list. I think
> > it won't be enough to just change the locking mode because with the
> > current patch strategy during attach, we will be first taking locks
> > for child tables of current partition and then parent tables which can
> > pose deadlock hazards.
> > > The columns list validation also needs to be done when we change
> > publication action.
> >
> I believe those issues should be solved by adopting the same approach as
> the row-filtering patch, right?
>

Right.

>
> > Some other miscellaneous comments:
> > =============================
> > *
> > In get_rel_sync_entry(), the handling for partitioned tables doesn't
> > seem to be correct. It can publish a different set of columns based on
> > the order of publications specified in the subscription.
> >
> > For example:
> > ----
> > create table parent (a int, b int, c int) partition by range (a);
> > create table test_part1 (like parent);
> > alter table parent attach partition test_part1 for values from (1) to (10);
> >
> > create publication pub for table parent(a) with (PUBLISH_VIA_PARTITION_ROOT);
> > create publication pub2 for table test_part1(b);
> > ---
> >
> > Now, depending on the order of publications in the list while defining
> > subscription, the column list will change
> > ----
> > create subscription sub connection 'port=10000 dbname=postgres'
> > publication pub, pub2;
> >
> > For the above, column list will be: (a)
> >
> > create subscription sub connection 'port=10000 dbname=postgres'
> > publication pub2, pub;
> >
> > For this one, the column list will be: (a, b)
> > ----
> >
> > To avoid this, the column list should be computed based on the final
> > publish_as_relid as we are doing for the row filter.
> >
>
> Hmm, yeah. That seems like a genuine problem - it should not depend on
> the order of publications in the subscription, I guess.
>
> But is it an issue in the patch? Isn't that a pre-existing issue? AFAICS
> the problem is that we initialize publish_as_relid=relid before the loop
> over publications, and then just update it. So the first iteration
> starts with relid, but the second iteration ends with whatever value is
> set by the first iteration (e.g. the root).
>
> So with the example you posted, we start with
>
>   publish_as_relid = relid = test_part1
>
> but then if the first publication is pubviaroot=true, we update it to
> parent. And in the second iteration, we fail to find the column filter,
> because "parent" (publish_as_relid) is not part of the pub2.
>
> If we do it in the other order, we leave the publish_as_relid value as
> is (and find the filter), and then update it in the second iteration
> (and find the column filter too).
>
> Now, this can be resolved by re-calculating the publish_as_relid from
> scratch in each iteration (start with relid, then maybe update it). But
> that's just half the story - the issue is there even without column
> filters. Consider this example:
>
> create table t (a int, b int, c int) partition by range (a);
>
> create table t_1 partition of t for values from (1) to (10)
>        partition by range (a);
>
> create table t_2 partition of t_1 for values from (1) to (10);
>
> create publication pub1 for table t(a)
>   with (PUBLISH_VIA_PARTITION_ROOT);
>
> create publication pub2 for table t_1(a)
>   with (PUBLISH_VIA_PARTITION_ROOT);
>
>
> Now, is you change subscribe to "pub1, pub2" and "pub2, pub1", we'll end
> up with different publish_as_relid values (t or t_1). Which seems like
> the same ambiguity issue.
>

I think we should fix this existing problem by always using the
top-most table as publish_as_relid. Basically, we can check, if the
existing publish_as_relid is an ancestor of a new rel that is going to
replace it then we shouldn't replace it. However, I think even if we
fix the existing problem, we will still have the order problem for the
column filter patch, and to avoid that instead of fetching column
filters in the publication loop, we should use the final
publish_as_relid. I think it will have another problem as well if we
don't use final publish_as_relid which is that sometimes when we
should not use any filter (say when pubviaroot is true and that
publication has root partitioned table which has no filter) as per our
rule of filters for a partitioned table, it can still use some filter
from the non-root table.

>
> > *
> > Fetching column filter info in tablesync.c is quite expensive. It
> > seems to be using four round-trips to get the complete info whereas
> > for row-filter we use just one round trip. I think we should try to
> > get both row filter and column filter info in just one round trip.
> >
>
> Maybe, but I really don't think this is an issue.
>

I am not sure but it might matter for small tables. Leaving aside the
performance issue, I think the current way will get the wrong column
list in many cases: (a) The ALL TABLES IN SCHEMA case handling won't
work for partitioned tables when the partitioned table is part of one
schema and partition table is part of another schema. (b) The handling
of partition tables in other cases will fetch incorrect lists as it
tries to fetch the column list of all the partitions in the hierarchy.

One of my colleagues has even tested these cases both for column
filters and row filters and we find the behavior of row filter is okay
whereas for column filter it uses the wrong column list. We will share
the tests and results with you in a later email. We are trying to
unify the column filter queries with row filter to make their behavior
the same and will share the findings once it is done. I hope if we are
able to achieve this that we will reduce the chances of bugs in this
area.

Note: I think the first two patches for tests are not required after
commit ceb57afd3c.

-- 
With Regards,
Amit Kapila.

  151. RE: Column Filtering in Logical Replication

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2022-03-09T10:12:01Z

    On Wednesday, March 9, 2022 6:04 PM Amit Kapila <amit.kapila16@gmail.com>
> On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
> >
> > On 3/4/22 11:42, Amit Kapila wrote:
> >
> > > *
> > > Fetching column filter info in tablesync.c is quite expensive. It
> > > seems to be using four round-trips to get the complete info whereas
> > > for row-filter we use just one round trip. I think we should try to
> > > get both row filter and column filter info in just one round trip.
> > >
> >
> > Maybe, but I really don't think this is an issue.
> >
> 
> I am not sure but it might matter for small tables. Leaving aside the
> performance issue, I think the current way will get the wrong column list in
> many cases: (a) The ALL TABLES IN SCHEMA case handling won't work for
> partitioned tables when the partitioned table is part of one schema and
> partition table is part of another schema. (b) The handling of partition tables in
> other cases will fetch incorrect lists as it tries to fetch the column list of all the
> partitions in the hierarchy.
> 
> One of my colleagues has even tested these cases both for column filters and
> row filters and we find the behavior of row filter is okay whereas for column
> filter it uses the wrong column list. We will share the tests and results with you
> in a later email. We are trying to unify the column filter queries with row filter to
> make their behavior the same and will share the findings once it is done. I hope
> if we are able to achieve this that we will reduce the chances of bugs in this area.
> 
> Note: I think the first two patches for tests are not required after commit
> ceb57afd3c.

Hi,

Here are some tests and results about the table sync query of
column filter patch and row filter.

1) multiple publications which publish schema of parent table and partition.
----pub
create schema s1;
create table s1.t (a int, b int, c int) partition by range (a);
create table t_1 partition of s1.t for values from (1) to (10);
create publication pub1 for all tables in schema s1;
create publication pub2 for table t_1(b);

----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub1, pub2;

When doing table sync for 't_1', the column list will be (b). I think it should
be no filter because table t_1 is also published via ALL TABLES IN SCHMEA
publication.

For Row Filter, it will use no filter for this case.


2) one publication publishes both parent and child
----pub
create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
       partition by range (a);
create table t_2 partition of t_1 for values from (1) to (10);

create publication pub2 for table t_1(a), t_2
  with (PUBLISH_VIA_PARTITION_ROOT);

----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;

When doing table sync for table 't_1', it has no column list. I think the
expected column list is (a).

For Row Filter, it will use the row filter of the top most parent table(t_1) in
this case.


3) one publication publishes both parent and child
----pub
create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
       partition by range (a);
create table t_2 partition of t_1 for values from (1) to (10);

create publication pub2 for table t_1(a), t_2(b)
  with (PUBLISH_VIA_PARTITION_ROOT);

----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;

When doing table sync for table 't_1', the column list would be (a, b). I think
the expected column list is (a).

For Row Filter, it will use the row filter of the top most parent table(t_1) in
this case.

Best regards,
Hou zj

  152. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-09T13:53:12Z

    On 3/9/22 11:03, Amit Kapila wrote:
> ...
>> Hmm, yeah. That seems like a genuine problem - it should not depend on
>> the order of publications in the subscription, I guess.
>>
>> But is it an issue in the patch? Isn't that a pre-existing issue? AFAICS
>> the problem is that we initialize publish_as_relid=relid before the loop
>> over publications, and then just update it. So the first iteration
>> starts with relid, but the second iteration ends with whatever value is
>> set by the first iteration (e.g. the root).
>>
>> So with the example you posted, we start with
>>
>>   publish_as_relid = relid = test_part1
>>
>> but then if the first publication is pubviaroot=true, we update it to
>> parent. And in the second iteration, we fail to find the column filter,
>> because "parent" (publish_as_relid) is not part of the pub2.
>>
>> If we do it in the other order, we leave the publish_as_relid value as
>> is (and find the filter), and then update it in the second iteration
>> (and find the column filter too).
>>
>> Now, this can be resolved by re-calculating the publish_as_relid from
>> scratch in each iteration (start with relid, then maybe update it). But
>> that's just half the story - the issue is there even without column
>> filters. Consider this example:
>>
>> create table t (a int, b int, c int) partition by range (a);
>>
>> create table t_1 partition of t for values from (1) to (10)
>>        partition by range (a);
>>
>> create table t_2 partition of t_1 for values from (1) to (10);
>>
>> create publication pub1 for table t(a)
>>   with (PUBLISH_VIA_PARTITION_ROOT);
>>
>> create publication pub2 for table t_1(a)
>>   with (PUBLISH_VIA_PARTITION_ROOT);
>>
>>
>> Now, is you change subscribe to "pub1, pub2" and "pub2, pub1", we'll end
>> up with different publish_as_relid values (t or t_1). Which seems like
>> the same ambiguity issue.
>>
> 
> I think we should fix this existing problem by always using the
> top-most table as publish_as_relid. Basically, we can check, if the
> existing publish_as_relid is an ancestor of a new rel that is going to
> replace it then we shouldn't replace it.

Right, using the topmost relid from all publications seems like the
correct solution.

> However, I think even if we
> fix the existing problem, we will still have the order problem for the
> column filter patch, and to avoid that instead of fetching column
> filters in the publication loop, we should use the final
> publish_as_relid. I think it will have another problem as well if we
> don't use final publish_as_relid which is that sometimes when we
> should not use any filter (say when pubviaroot is true and that
> publication has root partitioned table which has no filter) as per our
> rule of filters for a partitioned table, it can still use some filter
> from the non-root table.
> 

Yeah, the current behavior is just a consequence of how we determine
publish_as_relid now. If we rework that, we should first determine the
relid and then fetch the filter only for that single rel.

>>
>>> *
>>> Fetching column filter info in tablesync.c is quite expensive. It
>>> seems to be using four round-trips to get the complete info whereas
>>> for row-filter we use just one round trip. I think we should try to
>>> get both row filter and column filter info in just one round trip.
>>>
>>
>> Maybe, but I really don't think this is an issue.
>>
> 
> I am not sure but it might matter for small tables. Leaving aside the
> performance issue, I think the current way will get the wrong column
> list in many cases: (a) The ALL TABLES IN SCHEMA case handling won't
> work for partitioned tables when the partitioned table is part of one
> schema and partition table is part of another schema. (b) The handling
> of partition tables in other cases will fetch incorrect lists as it
> tries to fetch the column list of all the partitions in the hierarchy.
> 
> One of my colleagues has even tested these cases both for column
> filters and row filters and we find the behavior of row filter is okay
> whereas for column filter it uses the wrong column list. We will share
> the tests and results with you in a later email. We are trying to
> unify the column filter queries with row filter to make their behavior
> the same and will share the findings once it is done. I hope if we are
> able to achieve this that we will reduce the chances of bugs in this
> area.
> 

OK, I'll take a look at that email.

> Note: I think the first two patches for tests are not required after
> commit ceb57afd3c.
> 

Right. Will remove.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  153. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-10T03:09:46Z

    On Wed, Mar 9, 2022 at 3:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>
> > OK, I reworked this to do the same thing as the row filtering patch.
> >
>
> Thanks, I'll check this.
>

Some assorted comments:
=====================
1. We don't need to send a column list for the old tuple in case of an
update (similar to delete). It is not required to apply a column
filter for those cases because we ensure that RI must be part of the
column list for updates and deletes.
2.
+ /*
+ * Check if all columns referenced in the column filter are part of
+ * the REPLICA IDENTITY index or not.

I think this comment is reverse. The rule we follow here is that
attributes that are part of RI must be there in a specified column
list. This is used at two places in the patch.
3. get_rel_sync_entry()
{
/* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
...
}

Similar to the row filter, I think we need to use
entry->cache_expr_cxt to allocate this. There are other usages of
CacheMemoryContext in this part of the code but I think those need to
be also changed and we can do that as a separate patch. If we do the
suggested change then we don't need to separately free columns.
4. I think we don't need the DDL changes in AtExecDropColumn. Instead,
we can change the dependency of columns to NORMAL during publication
commands.
5. There is a reference to check_publication_columns but that function
is removed from the patch.
6.
/*
* If we know everything is replicated and the row filter is invalid
* for update and delete, there is no point to check for other
* publications.
*/
if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;

/*
* If we know everything is replicated and the column filter is invalid
* for update and delete, there is no point to check for other
* publications.
*/
if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->cf_valid_for_update && !pubdesc->cf_valid_for_delete)
break;

Can we combine these two checks?

I feel this patch needs a more thorough review.

-- 
With Regards,
Amit Kapila.

  154. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-10T18:17:15Z

    On 3/9/22 11:12, houzj.fnst@fujitsu.com wrote:
> Hi,
> 
> Here are some tests and results about the table sync query of
> column filter patch and row filter.
> 
> 1) multiple publications which publish schema of parent table and partition.
> ----pub
> create schema s1;
> create table s1.t (a int, b int, c int) partition by range (a);
> create table t_1 partition of s1.t for values from (1) to (10);
> create publication pub1 for all tables in schema s1;
> create publication pub2 for table t_1(b);
> 
> ----sub
> - prepare tables
> CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub1, pub2;
> 
> When doing table sync for 't_1', the column list will be (b). I think it should
> be no filter because table t_1 is also published via ALL TABLES IN SCHMEA
> publication.
> 
> For Row Filter, it will use no filter for this case.
> 
> 
> 2) one publication publishes both parent and child
> ----pub
> create table t (a int, b int, c int) partition by range (a);
> create table t_1 partition of t for values from (1) to (10)
>        partition by range (a);
> create table t_2 partition of t_1 for values from (1) to (10);
> 
> create publication pub2 for table t_1(a), t_2
>   with (PUBLISH_VIA_PARTITION_ROOT);
> 
> ----sub
> - prepare tables
> CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;
> 
> When doing table sync for table 't_1', it has no column list. I think the
> expected column list is (a).
> 
> For Row Filter, it will use the row filter of the top most parent table(t_1) in
> this case.
> 
> 
> 3) one publication publishes both parent and child
> ----pub
> create table t (a int, b int, c int) partition by range (a);
> create table t_1 partition of t for values from (1) to (10)
>        partition by range (a);
> create table t_2 partition of t_1 for values from (1) to (10);
> 
> create publication pub2 for table t_1(a), t_2(b)
>   with (PUBLISH_VIA_PARTITION_ROOT);
> 
> ----sub
> - prepare tables
> CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;
> 
> When doing table sync for table 't_1', the column list would be (a, b). I think
> the expected column list is (a).
> 
> For Row Filter, it will use the row filter of the top most parent table(t_1) in
> this case.
> 

Attached is an updated patch version, addressing all of those issues.

0001 is a bugfix, reworking how we calculate publish_as_relid. The old
approach was unstable with multiple publications, giving different
results depending on order of the publications. This should be
backpatched into PG13 where publish_via_partition_root was introduced, I
think.

0002 is the main patch, merging the changes proposed by Peter and fixing
the issues reported here. In most cases this means adopting the code
used for row filters, and perhaps simplifying it a bit.


But I also tried to implement a row-filter test for 0001, and I'm not
sure I understand the behavior I observe. Consider this:

-- a chain of 3 partitions (on both publisher and subscriber)
CREATE TABLE test_part_rf (a int primary key, b int, c int)
       PARTITION BY LIST (a);

CREATE TABLE test_part_rf_1
       PARTITION OF test_part_rf FOR VALUES IN (1,2,3,4,5)
       PARTITION BY LIST (a);

CREATE TABLE test_part_rf_2
       PARTITION OF test_part_rf_1 FOR VALUES IN (1,2,3,4,5);

-- initial data
INSERT INTO test_part_rf VALUES (1, 5, 100);
INSERT INTO test_part_rf VALUES (2, 15, 200);

-- two publications, each adding a different partition
CREATE PUBLICATION test_pub_part_1 FOR TABLE test_part_rf_1
 WHERE (b < 10) WITH (publish_via_partition_root);

CREATE PUBLICATION test_pub_part_2 FOR TABLE test_part_rf_2
 WHERE (b > 10) WITH (publish_via_partition_root);

-- now create the subscription (also try opposite ordering)
CREATE SUBSCRIPTION test_part_sub CONNECTION '...'
       PUBLICATION test_pub_part_1, test_pub_part_2;

-- wait for sync

-- inert some more data
INSERT INTO test_part_rf VALUES (3, 6, 300);
INSERT INTO test_part_rf VALUES (4, 16, 400);

-- wait for catchup

Now, based on the discussion here, my expectation is that we'll use the
row filter from the top-most ancestor in any publication, which in this
case is test_part_rf_1. Hence the filter should be (b < 10).

So I'd expect these rows to be replicated:

1,5,100
3,6,300

But that's not what I get, unfortunately. I get different results,
depending on the order of publications:

1) test_pub_part_1, test_pub_part_2

1|5|100
2|15|200
3|6|300
4|16|400

2) test_pub_part_2, test_pub_part_1

3|6|300
4|16|400

That seems pretty bizarre, because it either means we're not enforcing
any filter or some strange combination of filters (notice that for (2)
we skip/replicate rows matching either filter).

I have to be missing something important, but this seems confusing.
There's a patch adding a simple test case to 028_row_filter.sql (named
.txt, so as not to confuse cfbot).


FWIW I'm not convinced applying just the filters (both row and column)
is the right approach. It might be OK for a single publication, but with
multiple publications not so much. If you list multiple publications for
a subscription, it seems natural to expect a union of all the data, a
bit as if there were multiple subscriptions. But what you actually get
is some subset, depending on what other relations the other publications
include.

Of course, this only happens if the publications include different
ancestors. If all include the same ancestor, everything works fine and
you get the "union" of data.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  155. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-10T18:20:10Z

    On 3/9/22 10:20, Peter Eisentraut wrote:
> 
> On 07.03.22 16:18, Tomas Vondra wrote:
>> AFAICS these issues should be resolved by the adoption of the row-filter
>> approach (i.e. it should fail the same way as for row filter).
> 
> The first two patches (additional testing for row filtering feature)
> look okay to me.
> 
> Attached is a fixup patch for your main feature patch (the third one).
> 
> It's a bit of code and documentation cleanup, but mainly I removed the
> term "column filter" from the patch.  Half the code was using "column
> list" or similar and half the code "column filter", which was confusing.
>  Also, there seemed to be a bit of copy-and-pasting from row-filter code
> going on, with some code comments not quite sensible, so I rewrote some
> of them.  Also some code used "rf" and "cf" symbols which were a bit
> hard to tell apart.  A few more letters can increase readability.
> 
> Note in publicationcmds.c OpenTableList() the wrong if condition was used.
> 

Thanks, I've merged these changes into the patch.

> I'm still confused about the intended replica identity handling.  This
> patch still checks whether the column list contains the replica identity
> at DDL time.  And then it also checks at execution time.  I thought the
> latest understanding was that the DDL-time checking would be removed.  I
> think it's basically useless now, since as the test cases show, you can
> subvert those checks by altering the replica identity later.

Are you sure? Which part of the patch does that? AFAICS we only do those
checks in CheckCmdReplicaIdentity now, but maybe I'm missing something.
Are you sure you're not looking at some older patch version?


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  156. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-10T19:10:01Z

    
On 3/10/22 19:17, Tomas Vondra wrote:
> On 3/9/22 11:12, houzj.fnst@fujitsu.com wrote:
>> Hi,
>>
>> Here are some tests and results about the table sync query of
>> column filter patch and row filter.
>>
>> 1) multiple publications which publish schema of parent table and partition.
>> ----pub
>> create schema s1;
>> create table s1.t (a int, b int, c int) partition by range (a);
>> create table t_1 partition of s1.t for values from (1) to (10);
>> create publication pub1 for all tables in schema s1;
>> create publication pub2 for table t_1(b);
>>
>> ----sub
>> - prepare tables
>> CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub1, pub2;
>>
>> When doing table sync for 't_1', the column list will be (b). I think it should
>> be no filter because table t_1 is also published via ALL TABLES IN SCHMEA
>> publication.
>>
>> For Row Filter, it will use no filter for this case.
>>
>>
>> 2) one publication publishes both parent and child
>> ----pub
>> create table t (a int, b int, c int) partition by range (a);
>> create table t_1 partition of t for values from (1) to (10)
>>        partition by range (a);
>> create table t_2 partition of t_1 for values from (1) to (10);
>>
>> create publication pub2 for table t_1(a), t_2
>>   with (PUBLISH_VIA_PARTITION_ROOT);
>>
>> ----sub
>> - prepare tables
>> CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;
>>
>> When doing table sync for table 't_1', it has no column list. I think the
>> expected column list is (a).
>>
>> For Row Filter, it will use the row filter of the top most parent table(t_1) in
>> this case.
>>
>>
>> 3) one publication publishes both parent and child
>> ----pub
>> create table t (a int, b int, c int) partition by range (a);
>> create table t_1 partition of t for values from (1) to (10)
>>        partition by range (a);
>> create table t_2 partition of t_1 for values from (1) to (10);
>>
>> create publication pub2 for table t_1(a), t_2(b)
>>   with (PUBLISH_VIA_PARTITION_ROOT);
>>
>> ----sub
>> - prepare tables
>> CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;
>>
>> When doing table sync for table 't_1', the column list would be (a, b). I think
>> the expected column list is (a).
>>
>> For Row Filter, it will use the row filter of the top most parent table(t_1) in
>> this case.
>>
> 
> Attached is an updated patch version, addressing all of those issues.
> 
> 0001 is a bugfix, reworking how we calculate publish_as_relid. The old
> approach was unstable with multiple publications, giving different
> results depending on order of the publications. This should be
> backpatched into PG13 where publish_via_partition_root was introduced, I
> think.
> 
> 0002 is the main patch, merging the changes proposed by Peter and fixing
> the issues reported here. In most cases this means adopting the code
> used for row filters, and perhaps simplifying it a bit.
> 
> 
> But I also tried to implement a row-filter test for 0001, and I'm not
> sure I understand the behavior I observe. Consider this:
> 
> -- a chain of 3 partitions (on both publisher and subscriber)
> CREATE TABLE test_part_rf (a int primary key, b int, c int)
>        PARTITION BY LIST (a);
> 
> CREATE TABLE test_part_rf_1
>        PARTITION OF test_part_rf FOR VALUES IN (1,2,3,4,5)
>        PARTITION BY LIST (a);
> 
> CREATE TABLE test_part_rf_2
>        PARTITION OF test_part_rf_1 FOR VALUES IN (1,2,3,4,5);
> 
> -- initial data
> INSERT INTO test_part_rf VALUES (1, 5, 100);
> INSERT INTO test_part_rf VALUES (2, 15, 200);
> 
> -- two publications, each adding a different partition
> CREATE PUBLICATION test_pub_part_1 FOR TABLE test_part_rf_1
>  WHERE (b < 10) WITH (publish_via_partition_root);
> 
> CREATE PUBLICATION test_pub_part_2 FOR TABLE test_part_rf_2
>  WHERE (b > 10) WITH (publish_via_partition_root);
> 
> -- now create the subscription (also try opposite ordering)
> CREATE SUBSCRIPTION test_part_sub CONNECTION '...'
>        PUBLICATION test_pub_part_1, test_pub_part_2;
> 
> -- wait for sync
> 
> -- inert some more data
> INSERT INTO test_part_rf VALUES (3, 6, 300);
> INSERT INTO test_part_rf VALUES (4, 16, 400);
> 
> -- wait for catchup
> 
> Now, based on the discussion here, my expectation is that we'll use the
> row filter from the top-most ancestor in any publication, which in this
> case is test_part_rf_1. Hence the filter should be (b < 10).
> 
> So I'd expect these rows to be replicated:
> 
> 1,5,100
> 3,6,300
> 
> But that's not what I get, unfortunately. I get different results,
> depending on the order of publications:
> 
> 1) test_pub_part_1, test_pub_part_2
> 
> 1|5|100
> 2|15|200
> 3|6|300
> 4|16|400
> 
> 2) test_pub_part_2, test_pub_part_1
> 
> 3|6|300
> 4|16|400
> 
> That seems pretty bizarre, because it either means we're not enforcing
> any filter or some strange combination of filters (notice that for (2)
> we skip/replicate rows matching either filter).
> 
> I have to be missing something important, but this seems confusing.
> There's a patch adding a simple test case to 028_row_filter.sql (named
> .txt, so as not to confuse cfbot).
> 

FWIW I think the reason is pretty simple - pgoutput_row_filter_init is
broken. It assumes you can just do this

rftuple = SearchSysCache2(PUBLICATIONRELMAP,
                          ObjectIdGetDatum(entry->publish_as_relid),
                          ObjectIdGetDatum(pub->oid));

if (HeapTupleIsValid(rftuple))
{
    /* Null indicates no filter. */
    rfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
                              Anum_pg_publication_rel_prqual,
                              &pub_no_filter);
}
else
{
    pub_no_filter = true;
}


and pub_no_filter=true means there's no filter at all. Which is
nonsense, because we're using publish_as_relid here - the publication
may not include this particular ancestor, in which case we need to just
ignore this publication.

So yeah, this needs to be reworked.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  157. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-10T19:14:37Z

    
On 3/10/22 04:09, Amit Kapila wrote:
> On Wed, Mar 9, 2022 at 3:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
>> <tomas.vondra@enterprisedb.com> wrote:
>>
>>> OK, I reworked this to do the same thing as the row filtering patch.
>>>
>>
>> Thanks, I'll check this.
>>
> 
> Some assorted comments:
> =====================
> 1. We don't need to send a column list for the old tuple in case of an
> update (similar to delete). It is not required to apply a column
> filter for those cases because we ensure that RI must be part of the
> column list for updates and deletes.

I'm not sure which part of the code does this refer to?

> 2.
> + /*
> + * Check if all columns referenced in the column filter are part of
> + * the REPLICA IDENTITY index or not.
> 
> I think this comment is reverse. The rule we follow here is that
> attributes that are part of RI must be there in a specified column
> list. This is used at two places in the patch.

Yeah, you're right. Will fix.

> 3. get_rel_sync_entry()
> {
> /* XXX is there a danger of memory leak here? beware */
> + oldctx = MemoryContextSwitchTo(CacheMemoryContext);
> + for (int i = 0; i < nelems; i++)
> ...
> }
> 
> Similar to the row filter, I think we need to use
> entry->cache_expr_cxt to allocate this. There are other usages of
> CacheMemoryContext in this part of the code but I think those need to
> be also changed and we can do that as a separate patch. If we do the
> suggested change then we don't need to separately free columns.

I agree a shorter-lived context would be better than CacheMemoryContext,
but "expr" seems to indicate it's for the expression only, so maybe we
should rename that. But do we really want a memory context for every
single entry?

> 4. I think we don't need the DDL changes in AtExecDropColumn. Instead,
> we can change the dependency of columns to NORMAL during publication
> commands.

I'll think about that.

> 5. There is a reference to check_publication_columns but that function
> is removed from the patch.

Right, will fix.

> 6.
> /*
> * If we know everything is replicated and the row filter is invalid
> * for update and delete, there is no point to check for other
> * publications.
> */
> if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
> pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
> !pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
> break;
> 
> /*
> * If we know everything is replicated and the column filter is invalid
> * for update and delete, there is no point to check for other
> * publications.
> */
> if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
> pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
> !pubdesc->cf_valid_for_update && !pubdesc->cf_valid_for_delete)
> break;
> 
> Can we combine these two checks?
> 

I was worried it'd get too complex / hard to understand, but I'll think
about maybe simplifying the conditions a bit.

> I feel this patch needs a more thorough review.
> 

I won't object to more review, of course.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  158. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-11T01:56:41Z

    On 3/10/22 20:10, Tomas Vondra wrote:
> 
> 
> On 3/10/22 19:17, Tomas Vondra wrote:
>> On 3/9/22 11:12, houzj.fnst@fujitsu.com wrote:
>>> Hi,
>>>
>>> Here are some tests and results about the table sync query of
>>> column filter patch and row filter.
>>>
>>> 1) multiple publications which publish schema of parent table and partition.
>>> ----pub
>>> create schema s1;
>>> create table s1.t (a int, b int, c int) partition by range (a);
>>> create table t_1 partition of s1.t for values from (1) to (10);
>>> create publication pub1 for all tables in schema s1;
>>> create publication pub2 for table t_1(b);
>>>
>>> ----sub
>>> - prepare tables
>>> CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub1, pub2;
>>>
>>> When doing table sync for 't_1', the column list will be (b). I think it should
>>> be no filter because table t_1 is also published via ALL TABLES IN SCHMEA
>>> publication.
>>>
>>> For Row Filter, it will use no filter for this case.
>>>
>>>
>>> 2) one publication publishes both parent and child
>>> ----pub
>>> create table t (a int, b int, c int) partition by range (a);
>>> create table t_1 partition of t for values from (1) to (10)
>>>        partition by range (a);
>>> create table t_2 partition of t_1 for values from (1) to (10);
>>>
>>> create publication pub2 for table t_1(a), t_2
>>>   with (PUBLISH_VIA_PARTITION_ROOT);
>>>
>>> ----sub
>>> - prepare tables
>>> CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;
>>>
>>> When doing table sync for table 't_1', it has no column list. I think the
>>> expected column list is (a).
>>>
>>> For Row Filter, it will use the row filter of the top most parent table(t_1) in
>>> this case.
>>>
>>>
>>> 3) one publication publishes both parent and child
>>> ----pub
>>> create table t (a int, b int, c int) partition by range (a);
>>> create table t_1 partition of t for values from (1) to (10)
>>>        partition by range (a);
>>> create table t_2 partition of t_1 for values from (1) to (10);
>>>
>>> create publication pub2 for table t_1(a), t_2(b)
>>>   with (PUBLISH_VIA_PARTITION_ROOT);
>>>
>>> ----sub
>>> - prepare tables
>>> CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;
>>>
>>> When doing table sync for table 't_1', the column list would be (a, b). I think
>>> the expected column list is (a).
>>>
>>> For Row Filter, it will use the row filter of the top most parent table(t_1) in
>>> this case.
>>>
>>
>> Attached is an updated patch version, addressing all of those issues.
>>
>> 0001 is a bugfix, reworking how we calculate publish_as_relid. The old
>> approach was unstable with multiple publications, giving different
>> results depending on order of the publications. This should be
>> backpatched into PG13 where publish_via_partition_root was introduced, I
>> think.
>>
>> 0002 is the main patch, merging the changes proposed by Peter and fixing
>> the issues reported here. In most cases this means adopting the code
>> used for row filters, and perhaps simplifying it a bit.
>>
>>
>> But I also tried to implement a row-filter test for 0001, and I'm not
>> sure I understand the behavior I observe. Consider this:
>>
>> -- a chain of 3 partitions (on both publisher and subscriber)
>> CREATE TABLE test_part_rf (a int primary key, b int, c int)
>>        PARTITION BY LIST (a);
>>
>> CREATE TABLE test_part_rf_1
>>        PARTITION OF test_part_rf FOR VALUES IN (1,2,3,4,5)
>>        PARTITION BY LIST (a);
>>
>> CREATE TABLE test_part_rf_2
>>        PARTITION OF test_part_rf_1 FOR VALUES IN (1,2,3,4,5);
>>
>> -- initial data
>> INSERT INTO test_part_rf VALUES (1, 5, 100);
>> INSERT INTO test_part_rf VALUES (2, 15, 200);
>>
>> -- two publications, each adding a different partition
>> CREATE PUBLICATION test_pub_part_1 FOR TABLE test_part_rf_1
>>  WHERE (b < 10) WITH (publish_via_partition_root);
>>
>> CREATE PUBLICATION test_pub_part_2 FOR TABLE test_part_rf_2
>>  WHERE (b > 10) WITH (publish_via_partition_root);
>>
>> -- now create the subscription (also try opposite ordering)
>> CREATE SUBSCRIPTION test_part_sub CONNECTION '...'
>>        PUBLICATION test_pub_part_1, test_pub_part_2;
>>
>> -- wait for sync
>>
>> -- inert some more data
>> INSERT INTO test_part_rf VALUES (3, 6, 300);
>> INSERT INTO test_part_rf VALUES (4, 16, 400);
>>
>> -- wait for catchup
>>
>> Now, based on the discussion here, my expectation is that we'll use the
>> row filter from the top-most ancestor in any publication, which in this
>> case is test_part_rf_1. Hence the filter should be (b < 10).
>>
>> So I'd expect these rows to be replicated:
>>
>> 1,5,100
>> 3,6,300
>>
>> But that's not what I get, unfortunately. I get different results,
>> depending on the order of publications:
>>
>> 1) test_pub_part_1, test_pub_part_2
>>
>> 1|5|100
>> 2|15|200
>> 3|6|300
>> 4|16|400
>>
>> 2) test_pub_part_2, test_pub_part_1
>>
>> 3|6|300
>> 4|16|400
>>
>> That seems pretty bizarre, because it either means we're not enforcing
>> any filter or some strange combination of filters (notice that for (2)
>> we skip/replicate rows matching either filter).
>>
>> I have to be missing something important, but this seems confusing.
>> There's a patch adding a simple test case to 028_row_filter.sql (named
>> .txt, so as not to confuse cfbot).
>>
> 
> FWIW I think the reason is pretty simple - pgoutput_row_filter_init is
> broken. It assumes you can just do this
> 
> rftuple = SearchSysCache2(PUBLICATIONRELMAP,
>                           ObjectIdGetDatum(entry->publish_as_relid),
>                           ObjectIdGetDatum(pub->oid));
> 
> if (HeapTupleIsValid(rftuple))
> {
>     /* Null indicates no filter. */
>     rfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
>                               Anum_pg_publication_rel_prqual,
>                               &pub_no_filter);
> }
> else
> {
>     pub_no_filter = true;
> }
> 
> 
> and pub_no_filter=true means there's no filter at all. Which is
> nonsense, because we're using publish_as_relid here - the publication
> may not include this particular ancestor, in which case we need to just
> ignore this publication.
> 
> So yeah, this needs to be reworked.
> 

I spent a bit of time looking at this, and I think a minor change in
get_rel_sync_entry() fixes this - it's enough to ensure rel_publications
only includes publications that actually include publish_as_relid.

But this does not address tablesync.c :-( That still copies everything,
because it decides to sync both rels (test_pub_part_1, test_pub_part_2),
with it's row filter. On older releases this would fail, because we'd
start two workers:

1) COPY public.test_part_rf_2 TO STDOUT

2) COPY (SELECT a, b, c FROM public.test_part_rf_1) TO STDOUT

And that ends up inserting date from test_part_rf_2 twice. But now we
end up doing this instead:

1) COPY (SELECT a, b, c FROM public.test_part_rf_1 WHERE (b < 10)) TO STDOUT

2) COPY (SELECT a, b, c FROM ONLY public.test_part_rf_2 WHERE (b > 10))
TO STDOUT

Which no longer conflicts, because those subsets are mutually exclusive
(due to how the filter is defined), so the sync succeeds.

But I find this really weird - I think it's reasonable to expect the
sync to produce the same result as if the data was inserted and
replicated, and this just violates that.

Shouldn't tablesync calculate a list of relations in a way that prevents
such duplicate / overlapping syncs? In any case, this sync issue looks
entirely unrelated to the column filtering patch.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  159. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-11T02:46:57Z

    On Fri, Mar 11, 2022 at 12:44 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/10/22 04:09, Amit Kapila wrote:
> > On Wed, Mar 9, 2022 at 3:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >>
> >> On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
> >> <tomas.vondra@enterprisedb.com> wrote:
> >>
> >>> OK, I reworked this to do the same thing as the row filtering patch.
> >>>
> >>
> >> Thanks, I'll check this.
> >>
> >
> > Some assorted comments:
> > =====================
> > 1. We don't need to send a column list for the old tuple in case of an
> > update (similar to delete). It is not required to apply a column
> > filter for those cases because we ensure that RI must be part of the
> > column list for updates and deletes.
>
> I'm not sure which part of the code does this refer to?
>

The below part:
@@ -464,11 +473,11 @@ logicalrep_write_update(StringInfo out,
TransactionId xid, Relation rel,
  pq_sendbyte(out, 'O'); /* old tuple follows */
  else
  pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
  }

I think here instead of columns, the patch needs to send NULL as it is
already doing in logicalrep_write_delete.

> > 2.
> > + /*
> > + * Check if all columns referenced in the column filter are part of
> > + * the REPLICA IDENTITY index or not.
> >
> > I think this comment is reverse. The rule we follow here is that
> > attributes that are part of RI must be there in a specified column
> > list. This is used at two places in the patch.
>
> Yeah, you're right. Will fix.
>
> > 3. get_rel_sync_entry()
> > {
> > /* XXX is there a danger of memory leak here? beware */
> > + oldctx = MemoryContextSwitchTo(CacheMemoryContext);
> > + for (int i = 0; i < nelems; i++)
> > ...
> > }
> >
> > Similar to the row filter, I think we need to use
> > entry->cache_expr_cxt to allocate this. There are other usages of
> > CacheMemoryContext in this part of the code but I think those need to
> > be also changed and we can do that as a separate patch. If we do the
> > suggested change then we don't need to separately free columns.
>
> I agree a shorter-lived context would be better than CacheMemoryContext,
> but "expr" seems to indicate it's for the expression only, so maybe we
> should rename that.
>

Yeah, we can do that. How about rel_entry_cxt or something like that?
The idea is that eventually, we should move a few other things of
RelSyncEntry like attrmap where we are using CacheMemoryContext under
this context.

> But do we really want a memory context for every
> single entry?
>

Any other better idea?

-- 
With Regards,
Amit Kapila.

  160. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-11T03:30:31Z

    On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> But this does not address tablesync.c :-( That still copies everything,
> because it decides to sync both rels (test_pub_part_1, test_pub_part_2),
> with it's row filter. On older releases this would fail, because we'd
> start two workers:
>

Yeah, this is because of the existing problem where we sync both rels
instead of one. We have fixed some similar existing problems earlier.
Hou-San has reported a similar case in another email [1].

>
> But I find this really weird - I think it's reasonable to expect the
> sync to produce the same result as if the data was inserted and
> replicated, and this just violates that.
>
> Shouldn't tablesync calculate a list of relations in a way that prevents
> such duplicate / overlapping syncs?
>

Yes, I think it is better to fix it separately than to fix it along
with row filter or column filter work.

>
In any case, this sync issue looks
> entirely unrelated to the column filtering patch.
>

Right.

[1] - https://www.postgresql.org/message-id/OS0PR01MB5716DC2982CC735FDE388804940B9%40OS0PR01MB5716.jpnprd01.prod.outlook.com

-- 
With Regards,
Amit Kapila.

  161. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-11T04:22:14Z

    On 3/11/22 03:46, Amit Kapila wrote:
> On Fri, Mar 11, 2022 at 12:44 AM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> On 3/10/22 04:09, Amit Kapila wrote:
>>> On Wed, Mar 9, 2022 at 3:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>>>
>>>> On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
>>>> <tomas.vondra@enterprisedb.com> wrote:
>>>>
>>>>> OK, I reworked this to do the same thing as the row filtering patch.
>>>>>
>>>>
>>>> Thanks, I'll check this.
>>>>
>>>
>>> Some assorted comments:
>>> =====================
>>> 1. We don't need to send a column list for the old tuple in case of an
>>> update (similar to delete). It is not required to apply a column
>>> filter for those cases because we ensure that RI must be part of the
>>> column list for updates and deletes.
>>
>> I'm not sure which part of the code does this refer to?
>>
> 
> The below part:
> @@ -464,11 +473,11 @@ logicalrep_write_update(StringInfo out,
> TransactionId xid, Relation rel,
>   pq_sendbyte(out, 'O'); /* old tuple follows */
>   else
>   pq_sendbyte(out, 'K'); /* old key follows */
> - logicalrep_write_tuple(out, rel, oldslot, binary);
> + logicalrep_write_tuple(out, rel, oldslot, binary, columns);
>   }
> 
> I think here instead of columns, the patch needs to send NULL as it is
> already doing in logicalrep_write_delete.
> 

Hmmm, yeah. In practice it doesn't really matter, because NULL means
"send all columns" so it actually relaxes the check. But we only send
the RI keys, which is a subset of the column filter. But will fix.

>>> 2.
>>> + /*
>>> + * Check if all columns referenced in the column filter are part of
>>> + * the REPLICA IDENTITY index or not.
>>>
>>> I think this comment is reverse. The rule we follow here is that
>>> attributes that are part of RI must be there in a specified column
>>> list. This is used at two places in the patch.
>>
>> Yeah, you're right. Will fix.
>>
>>> 3. get_rel_sync_entry()
>>> {
>>> /* XXX is there a danger of memory leak here? beware */
>>> + oldctx = MemoryContextSwitchTo(CacheMemoryContext);
>>> + for (int i = 0; i < nelems; i++)
>>> ...
>>> }
>>>
>>> Similar to the row filter, I think we need to use
>>> entry->cache_expr_cxt to allocate this. There are other usages of
>>> CacheMemoryContext in this part of the code but I think those need to
>>> be also changed and we can do that as a separate patch. If we do the
>>> suggested change then we don't need to separately free columns.
>>
>> I agree a shorter-lived context would be better than CacheMemoryContext,
>> but "expr" seems to indicate it's for the expression only, so maybe we
>> should rename that.
>>
> 
> Yeah, we can do that. How about rel_entry_cxt or something like that?
> The idea is that eventually, we should move a few other things of
> RelSyncEntry like attrmap where we are using CacheMemoryContext under
> this context.
> 

Yeah, rel_entry_cxt sounds fine I guess ...

>> But do we really want a memory context for every
>> single entry?
>>
> 
> Any other better idea?
> 

No, I think you're right - it'd be hard/impossible to keep track of all
the memory allocated for expression/estate. It'd be fine for the
columns, because that's just a bitmap, but not for the expressions.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  162. RE: Column Filtering in Logical Replication

    Wei Wang (Fujitsu) <wangw.fnst@fujitsu.com> — 2022-03-11T07:05:59Z

    On Fri, Mar 11, 2022 at 9:57 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>
Hi Tomas,
Thanks for your patches.

On Mon, Mar 9, 2022 at 9:53 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>On Wed, Mar 9, 2022 at 6:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>On Mon, Mar 7, 2022 at 11:18 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>>On Fri, Mar 4, 2022 at 6:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >>> Fetching column filter info in tablesync.c is quite expensive. It
> >>> seems to be using four round-trips to get the complete info whereas
> >>> for row-filter we use just one round trip. I think we should try to
> >>> get both row filter and column filter info in just one round trip.
> >>>
> >>
> >> Maybe, but I really don't think this is an issue.
> >>
> >
> > I am not sure but it might matter for small tables. Leaving aside the
> > performance issue, I think the current way will get the wrong column
> > list in many cases: (a) The ALL TABLES IN SCHEMA case handling won't
> > work for partitioned tables when the partitioned table is part of one
> > schema and partition table is part of another schema. (b) The handling
> > of partition tables in other cases will fetch incorrect lists as it
> > tries to fetch the column list of all the partitions in the hierarchy.
> >
> > One of my colleagues has even tested these cases both for column
> > filters and row filters and we find the behavior of row filter is okay
> > whereas for column filter it uses the wrong column list. We will share
> > the tests and results with you in a later email. We are trying to
> > unify the column filter queries with row filter to make their behavior
> > the same and will share the findings once it is done. I hope if we are
> > able to achieve this that we will reduce the chances of bugs in this
> > area.
> >
> 
> OK, I'll take a look at that email.
I tried to get both the column filters and the row filters with one SQL, but
it failed because I think the result is not easy to parse.

I noted that we use two SQLs to get column filters in the latest
patches(20220311). I think maybe we could use one SQL to get column filters to
reduce network cost. Like the SQL in the attachment.

Regards,
Wang wei

  163. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-11T09:52:19Z

    On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/10/22 20:10, Tomas Vondra wrote:
> >
> >
> > FWIW I think the reason is pretty simple - pgoutput_row_filter_init is
> > broken. It assumes you can just do this
> >
> > rftuple = SearchSysCache2(PUBLICATIONRELMAP,
> >                           ObjectIdGetDatum(entry->publish_as_relid),
> >                           ObjectIdGetDatum(pub->oid));
> >
> > if (HeapTupleIsValid(rftuple))
> > {
> >     /* Null indicates no filter. */
> >     rfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
> >                               Anum_pg_publication_rel_prqual,
> >                               &pub_no_filter);
> > }
> > else
> > {
> >     pub_no_filter = true;
> > }
> >
> >
> > and pub_no_filter=true means there's no filter at all. Which is
> > nonsense, because we're using publish_as_relid here - the publication
> > may not include this particular ancestor, in which case we need to just
> > ignore this publication.
> >
> > So yeah, this needs to be reworked.
> >
>
> I spent a bit of time looking at this, and I think a minor change in
> get_rel_sync_entry() fixes this - it's enough to ensure rel_publications
> only includes publications that actually include publish_as_relid.
>

Thanks for looking into this. I think in the first patch before
calling get_partition_ancestors() we need to ensure it is a partition
(the call expects that) and pubviaroot is true. I think it would be
good if we can avoid an additional call to get_partition_ancestors()
as it could be costly. I wonder why it is not sufficient to ensure
that publish_as_relid exists after ancestor in ancestors list before
assigning the ancestor to publish_as_relid? This only needs to be done
in case of (if (!publish)). I have not tried this so I could be wrong.

-- 
With Regards,
Amit Kapila.

  164. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-11T12:36:11Z

    
On 3/11/22 08:05, wangw.fnst@fujitsu.com wrote:
> On Fri, Mar 11, 2022 at 9:57 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>
> Hi Tomas,
> Thanks for your patches.
> 
> On Mon, Mar 9, 2022 at 9:53 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>> On Wed, Mar 9, 2022 at 6:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> On Mon, Mar 7, 2022 at 11:18 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>>> On Fri, Mar 4, 2022 at 6:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>>>> Fetching column filter info in tablesync.c is quite expensive. It
>>>>> seems to be using four round-trips to get the complete info whereas
>>>>> for row-filter we use just one round trip. I think we should try to
>>>>> get both row filter and column filter info in just one round trip.
>>>>>
>>>>
>>>> Maybe, but I really don't think this is an issue.
>>>>
>>>
>>> I am not sure but it might matter for small tables. Leaving aside the
>>> performance issue, I think the current way will get the wrong column
>>> list in many cases: (a) The ALL TABLES IN SCHEMA case handling won't
>>> work for partitioned tables when the partitioned table is part of one
>>> schema and partition table is part of another schema. (b) The handling
>>> of partition tables in other cases will fetch incorrect lists as it
>>> tries to fetch the column list of all the partitions in the hierarchy.
>>>
>>> One of my colleagues has even tested these cases both for column
>>> filters and row filters and we find the behavior of row filter is okay
>>> whereas for column filter it uses the wrong column list. We will share
>>> the tests and results with you in a later email. We are trying to
>>> unify the column filter queries with row filter to make their behavior
>>> the same and will share the findings once it is done. I hope if we are
>>> able to achieve this that we will reduce the chances of bugs in this
>>> area.
>>>
>>
>> OK, I'll take a look at that email.
> I tried to get both the column filters and the row filters with one SQL, but
> it failed because I think the result is not easy to parse.
> 
> I noted that we use two SQLs to get column filters in the latest
> patches(20220311). I think maybe we could use one SQL to get column filters to
> reduce network cost. Like the SQL in the attachment.
> 

I'll take a look. But as I said before - I very much prefer SQL that is
easy to understand, and I don't think the one extra round trip is an
issue during tablesync (which is a very rare action).


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  165. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-11T12:50:29Z

    
On 3/11/22 10:52, Amit Kapila wrote:
> On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> On 3/10/22 20:10, Tomas Vondra wrote:
>>>
>>>
>>> FWIW I think the reason is pretty simple - pgoutput_row_filter_init is
>>> broken. It assumes you can just do this
>>>
>>> rftuple = SearchSysCache2(PUBLICATIONRELMAP,
>>>                           ObjectIdGetDatum(entry->publish_as_relid),
>>>                           ObjectIdGetDatum(pub->oid));
>>>
>>> if (HeapTupleIsValid(rftuple))
>>> {
>>>     /* Null indicates no filter. */
>>>     rfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
>>>                               Anum_pg_publication_rel_prqual,
>>>                               &pub_no_filter);
>>> }
>>> else
>>> {
>>>     pub_no_filter = true;
>>> }
>>>
>>>
>>> and pub_no_filter=true means there's no filter at all. Which is
>>> nonsense, because we're using publish_as_relid here - the publication
>>> may not include this particular ancestor, in which case we need to just
>>> ignore this publication.
>>>
>>> So yeah, this needs to be reworked.
>>>
>>
>> I spent a bit of time looking at this, and I think a minor change in
>> get_rel_sync_entry() fixes this - it's enough to ensure rel_publications
>> only includes publications that actually include publish_as_relid.
>>
> 
> Thanks for looking into this. I think in the first patch before
> calling get_partition_ancestors() we need to ensure it is a partition
> (the call expects that) and pubviaroot is true.

Does the call really require that? Also, I'm not sure why we'd need to
look at pubviaroot - that's already considered earlier when calculating
publish_as_relid, here we just need to know the relationship of the two
OIDs (if one is ancestor/child of the other).

> I think it would be
> good if we can avoid an additional call to get_partition_ancestors()
> as it could be costly.

Maybe. OTOH we only should do this only very rarely anyway.

> I wonder why it is not sufficient to ensure
> that publish_as_relid exists after ancestor in ancestors list before
> assigning the ancestor to publish_as_relid? This only needs to be done
> in case of (if (!publish)). I have not tried this so I could be wrong.
> 

I'm not sure what exactly are you proposing. Maybe try coding it? That's
probably faster than trying to describe what the code might do ...


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  166. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-12T04:30:14Z

    On Fri, Mar 11, 2022 at 6:20 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/11/22 10:52, Amit Kapila wrote:
> > On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra
> > <tomas.vondra@enterprisedb.com> wrote:
> >>
> >> On 3/10/22 20:10, Tomas Vondra wrote:
> >>>
> >>>
> >>> FWIW I think the reason is pretty simple - pgoutput_row_filter_init is
> >>> broken. It assumes you can just do this
> >>>
> >>> rftuple = SearchSysCache2(PUBLICATIONRELMAP,
> >>>                           ObjectIdGetDatum(entry->publish_as_relid),
> >>>                           ObjectIdGetDatum(pub->oid));
> >>>
> >>> if (HeapTupleIsValid(rftuple))
> >>> {
> >>>     /* Null indicates no filter. */
> >>>     rfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
> >>>                               Anum_pg_publication_rel_prqual,
> >>>                               &pub_no_filter);
> >>> }
> >>> else
> >>> {
> >>>     pub_no_filter = true;
> >>> }
> >>>
> >>>
> >>> and pub_no_filter=true means there's no filter at all. Which is
> >>> nonsense, because we're using publish_as_relid here - the publication
> >>> may not include this particular ancestor, in which case we need to just
> >>> ignore this publication.
> >>>
> >>> So yeah, this needs to be reworked.
> >>>
> >>
> >> I spent a bit of time looking at this, and I think a minor change in
> >> get_rel_sync_entry() fixes this - it's enough to ensure rel_publications
> >> only includes publications that actually include publish_as_relid.
> >>
> >
> > Thanks for looking into this. I think in the first patch before
> > calling get_partition_ancestors() we need to ensure it is a partition
> > (the call expects that) and pubviaroot is true.
>
> Does the call really require that?
>

There may not be any harm but I have mentioned it because (a) the
comments atop get_partition_ancestors(...it should only be called when
it is known that the relation is a partition.) indicates the same; (b)
all existing callers seems to use it only for partitions.

> Also, I'm not sure why we'd need to
> look at pubviaroot - that's already considered earlier when calculating
> publish_as_relid, here we just need to know the relationship of the two
> OIDs (if one is ancestor/child of the other).
>

I thought of avoiding calling get_partition_ancestors when pubviaroot
is not set. It will unnecessary check the whole hierarchy for
partitions even when it is not required. I agree that this is not a
common code path but still felt why do it needlessly?

> > I think it would be
> > good if we can avoid an additional call to get_partition_ancestors()
> > as it could be costly.
>
> Maybe. OTOH we only should do this only very rarely anyway.
>
> > I wonder why it is not sufficient to ensure
> > that publish_as_relid exists after ancestor in ancestors list before
> > assigning the ancestor to publish_as_relid? This only needs to be done
> > in case of (if (!publish)). I have not tried this so I could be wrong.
> >
>
> I'm not sure what exactly are you proposing. Maybe try coding it? That's
> probably faster than trying to describe what the code might do ...
>

Okay, please find attached. I have done basic testing of this, if we
agree with this approach then this will require some more testing.

-- 
With Regards,
Amit Kapila.

  167. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-13T21:07:49Z

    On 3/12/22 05:30, Amit Kapila wrote:
>> ...
> 
> Okay, please find attached. I have done basic testing of this, if we
> agree with this approach then this will require some more testing.
> 

Thanks, the proposed changes seem like a clear improvement, so I've
added them, with some minor tweaks (mostly to comments).

I've also included the memory context rename (entry_changes to the
change proposed by Wang Wei, using a single SQL command in tablesync.

And I've renamed the per-entry memory context to entry_cxt, and used it
for the column list.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  168. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-14T09:53:44Z

    On Mon, Mar 14, 2022 at 2:37 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/12/22 05:30, Amit Kapila wrote:
> >> ...
> >
> > Okay, please find attached. I have done basic testing of this, if we
> > agree with this approach then this will require some more testing.
> >
>
> Thanks, the proposed changes seem like a clear improvement, so I've
> added them, with some minor tweaks (mostly to comments).
>

One minor point: Did you intentionally remove
list_free(rel_publications) before resetting the list from the second
patch? The memory for rel_publications is allocated in
TopTransactionContext, so a large transaction touching many relations
will only free this at end of the transaction which may not be a big
deal as we don't do this every time. We free this list a few lines
down in successful case so this appears slightly odd to me but I am
fine if you think it doesn't matter.


-- 
With Regards,
Amit Kapila.

  169. RE: Column Filtering in Logical Replication

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2022-03-14T11:12:25Z

    On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
> 
> On 3/12/22 05:30, Amit Kapila wrote:
> >> ...
> >
> > Okay, please find attached. I have done basic testing of this, if we
> > agree with this approach then this will require some more testing.
> >
> 
> Thanks, the proposed changes seem like a clear improvement, so I've
> added them, with some minor tweaks (mostly to comments).

Hi,

Thanks for updating the patches !
And sorry for the row filter bug caused by my mistake.

I looked at the two fixup patches. I am thinking would it be better if we
add one testcase for these two bugs? Maybe like the attachment.

(Attach the fixup patch to make the cfbot happy)

Best regards,
Hou zj

  170. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-14T12:05:24Z

    
On 3/14/22 10:53, Amit Kapila wrote:
> On Mon, Mar 14, 2022 at 2:37 AM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> On 3/12/22 05:30, Amit Kapila wrote:
>>>> ...
>>>
>>> Okay, please find attached. I have done basic testing of this, if we
>>> agree with this approach then this will require some more testing.
>>>
>>
>> Thanks, the proposed changes seem like a clear improvement, so I've
>> added them, with some minor tweaks (mostly to comments).
>>
> 
> One minor point: Did you intentionally remove
> list_free(rel_publications) before resetting the list from the second
> patch? The memory for rel_publications is allocated in
> TopTransactionContext, so a large transaction touching many relations
> will only free this at end of the transaction which may not be a big
> deal as we don't do this every time. We free this list a few lines
> down in successful case so this appears slightly odd to me but I am
> fine if you think it doesn't matter.

The removal was not intentional, but I don't think it's an issue exactly
because it's a tiny mount of memory and we'll release it at the end of
the transaction. Which should not take long.

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  171. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-14T12:11:58Z

    
On 3/14/22 12:12, houzj.fnst@fujitsu.com wrote:
> On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>
>> On 3/12/22 05:30, Amit Kapila wrote:
>>>> ...
>>>
>>> Okay, please find attached. I have done basic testing of this, if we
>>> agree with this approach then this will require some more testing.
>>>
>>
>> Thanks, the proposed changes seem like a clear improvement, so I've
>> added them, with some minor tweaks (mostly to comments).
> 
> Hi,
> 
> Thanks for updating the patches !
> And sorry for the row filter bug caused by my mistake.
> 
> I looked at the two fixup patches. I am thinking would it be better if we
> add one testcase for these two bugs? Maybe like the attachment.
> 

Yeah, a test would be nice - I'll take a look later.

Anyway, the fix does not address tablesync, as explained in [1]. I'm not
sure what to do about it - in principle, we could calculate which
relations to sync, and then eliminate "duplicates" (i.e. relations where
we are going to sync an ancestor).


regards

[1]
https://www.postgresql.org/message-id/822a8e40-287c-59ff-0ea9-35eb759f4fe6%40enterprisedb.com


-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  172. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-14T12:47:10Z

    On Mon, Mar 14, 2022 at 5:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/14/22 12:12, houzj.fnst@fujitsu.com wrote:
> > On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>
> Anyway, the fix does not address tablesync, as explained in [1]. I'm not
> sure what to do about it - in principle, we could calculate which
> relations to sync, and then eliminate "duplicates" (i.e. relations where
> we are going to sync an ancestor).
>

As mentioned in my previous email [1], this appears to be a base code
issue (even without row filter or column filter work), so it seems
better to deal with it separately. It has been reported separately as
well [2] where we found some similar issues.


[1] - https://www.postgresql.org/message-id/CAA4eK1LSb-xrvGEm3ShaRA%3DMkdii2d%2B4vqh9DGPvVDA%2BD9ibYw%40mail.gmail.com
[2] - https://www.postgresql.org/message-id/OS0PR01MB5716DC2982CC735FDE388804940B9@OS0PR01MB5716.jpnprd01.prod.outlook.com

-- 
With Regards,
Amit Kapila.

  173. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-14T13:32:53Z

    
On 3/14/22 13:47, Amit Kapila wrote:
> On Mon, Mar 14, 2022 at 5:42 PM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> On 3/14/22 12:12, houzj.fnst@fujitsu.com wrote:
>>> On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>
>> Anyway, the fix does not address tablesync, as explained in [1]. I'm not
>> sure what to do about it - in principle, we could calculate which
>> relations to sync, and then eliminate "duplicates" (i.e. relations where
>> we are going to sync an ancestor).
>>
> 
> As mentioned in my previous email [1], this appears to be a base code
> issue (even without row filter or column filter work), so it seems
> better to deal with it separately. It has been reported separately as
> well [2] where we found some similar issues.
> 

Right. I don't want to be waiting for that fix either, that'd block this
patch unnecessarily. If there are no other comments, I'll go ahead,
polish the existing patches a bit more and get them committed. We can
worry about this pre-existing issue later.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  174. RE: Column Filtering in Logical Replication

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-03-15T02:08:50Z

    On Mon, Mar 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
> 
> On 3/12/22 05:30, Amit Kapila wrote:
> >> ...
> >
> > Okay, please find attached. I have done basic testing of this, if we
> > agree with this approach then this will require some more testing.
> >
> 
> Thanks, the proposed changes seem like a clear improvement, so I've
> added them, with some minor tweaks (mostly to comments).
> 
> I've also included the memory context rename (entry_changes to the
> change proposed by Wang Wei, using a single SQL command in tablesync.
> 
> And I've renamed the per-entry memory context to entry_cxt, and used it
> for the column list.
> 

Thanks for your patch.
Here are some comments for column filter main patch (0003 patch).

1. doc/src/sgml/catalogs.sgml
@@ -6263,6 +6263,19 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
        Reference to schema
       </para></entry>
      </row>
+
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>prattrs</structfield> <type>int2vector</type>
+       (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+      </para>
+      <para>
+       This is an array of values that indicates which table columns are
+       part of the publication.  For example, a value of <literal>1 3</literal>
+       would mean that the first and the third table columns are published.
+       A null value indicates that all columns are published.
+      </para></entry>
+     </row>
     </tbody>
    </tgroup>
   </table>

This change was added to pg_publication_namespace view. I think it should be
added to pg_publication_rel view, right?

2. src/backend/replication/pgoutput/pgoutput.c
@@ -188,6 +202,7 @@ static EState *create_estate_for_relation(Relation rel);
 static void pgoutput_row_filter_init(PGOutputData *data,
 									 List *publications,
 									 RelationSyncEntry *entry);
+
 static bool pgoutput_row_filter_exec_expr(ExprState *state,
 										  ExprContext *econtext);
 static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,

Should we remove this change?

3. src/backend/commands/publicationcmds.c
+/*
+ * Check if all columns referenced in the column list are part of the
+ * REPLICA IDENTITY index or not.
+ *
+ * Returns true if any invalid column is found.
+ */

The comment for pub_collist_contains_invalid_column() seems wrong.  Should it be
"Check if all REPLICA IDENTITY columns are covered by the column list or not"?

4.
The patch doesn't allow delete and update operations if the target table uses
replica identity full and it is published with column list specified, even if
column list includes all columns in the table.

For example:
create table tbl (a int, b int, c int);
create publication pub for table tbl (a, b, c);
alter table tbl replica identity full;

postgres=# delete from tbl;
ERROR:  cannot delete from table "tbl"
DETAIL:  Column list used by the publication does not cover the replica identity.

Should we allow this case? I think it doesn't seem to cause harm.

5. 
Maybe we need some changes for tab-complete.c.

Regards,
Shi yu

  175. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-15T04:43:53Z

    On Mon, Mar 14, 2022 at 4:42 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
>
> On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
> >
> > On 3/12/22 05:30, Amit Kapila wrote:
> > >> ...
> > >
> > > Okay, please find attached. I have done basic testing of this, if we
> > > agree with this approach then this will require some more testing.
> > >
> >
> > Thanks, the proposed changes seem like a clear improvement, so I've
> > added them, with some minor tweaks (mostly to comments).
>
> Hi,
>
> Thanks for updating the patches !
> And sorry for the row filter bug caused by my mistake.
>
> I looked at the two fixup patches. I am thinking would it be better if we
> add one testcase for these two bugs? Maybe like the attachment.
>

Your tests look good to me. We might want to add some comments for
each test but I guess that can be done before committing. Tomas, it
seems you are planning to push these bug fixes, do let me know if you
want me to take care of these while you focus on the main patch? I
think the first patch needs to be backpatched till 13 and the second
one is for just HEAD.

-- 
With Regards,
Amit Kapila.

  176. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-15T04:46:19Z

    On Mon, Mar 14, 2022 at 7:02 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/14/22 13:47, Amit Kapila wrote:
> > On Mon, Mar 14, 2022 at 5:42 PM Tomas Vondra
> > <tomas.vondra@enterprisedb.com> wrote:
> >>
> >> On 3/14/22 12:12, houzj.fnst@fujitsu.com wrote:
> >>> On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
> >>
> >> Anyway, the fix does not address tablesync, as explained in [1]. I'm not
> >> sure what to do about it - in principle, we could calculate which
> >> relations to sync, and then eliminate "duplicates" (i.e. relations where
> >> we are going to sync an ancestor).
> >>
> >
> > As mentioned in my previous email [1], this appears to be a base code
> > issue (even without row filter or column filter work), so it seems
> > better to deal with it separately. It has been reported separately as
> > well [2] where we found some similar issues.
> >
>
> Right. I don't want to be waiting for that fix either, that'd block this
> patch unnecessarily. If there are no other comments, I'll go ahead,
> polish the existing patches a bit more and get them committed. We can
> worry about this pre-existing issue later.
>

I think the first two patches are ready to go. I haven't read the
latest version in detail but I have in mind that we want to get this
in for PG-15.

-- 
With Regards,
Amit Kapila.

  177. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-15T08:30:44Z

    
On 3/15/22 05:43, Amit Kapila wrote:
> On Mon, Mar 14, 2022 at 4:42 PM houzj.fnst@fujitsu.com
> <houzj.fnst@fujitsu.com> wrote:
>>
>> On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>>
>>> On 3/12/22 05:30, Amit Kapila wrote:
>>>>> ...
>>>>
>>>> Okay, please find attached. I have done basic testing of this, if we
>>>> agree with this approach then this will require some more testing.
>>>>
>>>
>>> Thanks, the proposed changes seem like a clear improvement, so I've
>>> added them, with some minor tweaks (mostly to comments).
>>
>> Hi,
>>
>> Thanks for updating the patches !
>> And sorry for the row filter bug caused by my mistake.
>>
>> I looked at the two fixup patches. I am thinking would it be better if we
>> add one testcase for these two bugs? Maybe like the attachment.
>>
> 
> Your tests look good to me. We might want to add some comments for
> each test but I guess that can be done before committing. Tomas, it
> seems you are planning to push these bug fixes, do let me know if you
> want me to take care of these while you focus on the main patch? I
> think the first patch needs to be backpatched till 13 and the second
> one is for just HEAD.
> 

Yeah, I plan to push the fixes later today. I'll polish them a bit
first, and merge the tests (shared by Hou zj) into the patches etc.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  178. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-16T11:27:43Z

    On Tue, Mar 15, 2022 at 7:38 AM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Mon, Mar 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>
> 3. src/backend/commands/publicationcmds.c
> +/*
> + * Check if all columns referenced in the column list are part of the
> + * REPLICA IDENTITY index or not.
> + *
> + * Returns true if any invalid column is found.
> + */
>
> The comment for pub_collist_contains_invalid_column() seems wrong.  Should it be
> "Check if all REPLICA IDENTITY columns are covered by the column list or not"?
>

On similar lines, I think errdetail for below messages need to be changed.
ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot update table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the
replica identity.")));
  else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
  ereport(ERROR,
  (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
  errmsg("cannot delete from table \"%s\"",
  RelationGetRelationName(rel)),
  errdetail("Column used in the publication WHERE expression is not
part of the replica identity.")));
+ else if (cmd == CMD_DELETE && !pubdesc.cols_valid_for_delete)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot delete from table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the
replica identity.")));

Some assorted comments:
========================
1. As mentioned previously as well[1], the change in ATExecDropColumn
is not required. Similarly, the change you seem to agree upon in
logicalrep_write_update[2] doesn't seem to be present.

2. I think the dependency handling in publication_set_table_columns()
has problems. While removing existing dependencies, it uses
PublicationRelationId as classId whereas while adding new dependencies
it uses PublicationRelRelationId as classId. This will create problems
while removing columns from table. For example,
postgres=# create table t1(c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# create publication pub1 for table t1(c1, c2);
CREATE PUBLICATION
postgres=# select * from pg_depend where classid = 6106 or refclassid
= 6106 or classid = 6104;
 classid | objid | objsubid | refclassid | refobjid | refobjsubid | deptype
---------+-------+----------+------------+----------+-------------+---------
    6106 | 16409 |        0 |       1259 |    16405 |           1 | a
    6106 | 16409 |        0 |       1259 |    16405 |           2 | a
    6106 | 16409 |        0 |       6104 |    16408 |           0 | a
    6106 | 16409 |        0 |       1259 |    16405 |           0 | a
(4 rows)

Till here everything is fine.

postgres=# Alter publication pub1 alter table t1 set columns(c2);
ALTER PUBLICATION
postgres=# select * from pg_depend where classid = 6106 or refclassid
= 6106 or classid = 6104;
 classid | objid | objsubid | refclassid | refobjid | refobjsubid | deptype
---------+-------+----------+------------+----------+-------------+---------
    6106 | 16409 |        0 |       1259 |    16405 |           1 | a
    6106 | 16409 |        0 |       1259 |    16405 |           2 | a
    6106 | 16409 |        0 |       6104 |    16408 |           0 | a
    6106 | 16409 |        0 |       1259 |    16405 |           0 | a
    6106 | 16409 |        0 |       1259 |    16405 |           2 | a
(5 rows)

Now without removing dependencies for columns 1 and 2, it added a new
dependency for column 2.

3.
@@ -930,8 +1054,24 @@ copy_table(Relation rel)
...
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
...
...
for (int i = 0; i < lrel.natts; i++)
{
appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
if (i < lrel.natts - 1)
appendStringInfoString(&cmd, ", ");
}

In the same function, we use two different styles to achieve the same
thing. I think it is better to use the same style (probably existing)
at both places for the sake of consistency.

4.
+  <para>
+   The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
+   the set of columns that are included in the publication.  If a column list
+   is specified, it must include the replica identity columns.
+  </para>

I think the second part holds true only for update/delete publications.

5.
+ * XXX Should this detect duplicate columns?
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns,
+   int *natts, AttrNumber **attrs)
{
...
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+    colname));
...
}

It seems we already detect duplicate columns in this function. So XXX
part of the comment doesn't seem to be required.

6.
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+    bool pubviaroot)

The second parameter is not used in this function. As noted in the
comments, I also think it is better to rename this. How about
ValidatePubColumnList?

7.
+ /*
+ * FIXME check pubactions vs. replica identity, to ensure the replica
+ * identity is included in the column list. Only do this for update
+ * and delete publications. See check_publication_columns.
+ *
+ * XXX This is needed because publish_via_partition_root may change,
+ * in which case the row filters may be invalid (e.g. with pvpr=false
+ * there must be no filter on partitioned tables).
+ */
+

This entire comment doesn't seem to be required.

8.
+publication_set_table_columns()
{
...
+ /* XXX "pub" is leaked here ??? */
...
}

It is not clear what this means?

9.
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ *
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
  * pub_obj is one of:
  *
  * TABLE table_name [, ...]
@@ -9869,6 +9878,32 @@ AlterPublicationStmt:
  n->action = AP_SetObjects;
  $$ = (Node *)n;
  }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '('
columnList ')'

The comments in gram.y indicates different rules than the actual implementation.

10.
+ *
+ * FIXME Do we need something similar for column filters?
  */
 enum RowFilterPubAction

I have thought about this point and it seems we don't need anything on
this front for this patch. We need the filter combining of
update/delete for row filter because if inserts have some column which
is not present in RI then during update filtering it can give an error
as the column won't be present in WAL log.

Now, the same problem won't be there for the column list/filter patch
because all the RI columns are there in the column list (for
update/delete) and we don't need to apply a column filter for old
tuples in either update or delete.

We can remove this FIXME.

11.
+ } /* loop all subscribed publications */
+
+}

No need for an empty line here.

[1] - https://www.postgresql.org/message-id/CAA4eK1K5pkrPT9z5TByUPptExian5c18g6GnfNf9Cr97QdPbjw%40mail.gmail.com
[2] - https://www.postgresql.org/message-id/43c15aa8-aa15-ca0f-40e4-3be68d98df05%40enterprisedb.com

-- 
With Regards,
Amit Kapila.

  179. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-16T17:24:58Z

    
On 3/15/22 09:30, Tomas Vondra wrote:
> 
> 
> On 3/15/22 05:43, Amit Kapila wrote:
>> On Mon, Mar 14, 2022 at 4:42 PM houzj.fnst@fujitsu.com
>> <houzj.fnst@fujitsu.com> wrote:
>>>
>>> On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>>>
>>>> On 3/12/22 05:30, Amit Kapila wrote:
>>>>>> ...
>>>>>
>>>>> Okay, please find attached. I have done basic testing of this, if we
>>>>> agree with this approach then this will require some more testing.
>>>>>
>>>>
>>>> Thanks, the proposed changes seem like a clear improvement, so I've
>>>> added them, with some minor tweaks (mostly to comments).
>>>
>>> Hi,
>>>
>>> Thanks for updating the patches !
>>> And sorry for the row filter bug caused by my mistake.
>>>
>>> I looked at the two fixup patches. I am thinking would it be better if we
>>> add one testcase for these two bugs? Maybe like the attachment.
>>>
>>
>> Your tests look good to me. We might want to add some comments for
>> each test but I guess that can be done before committing. Tomas, it
>> seems you are planning to push these bug fixes, do let me know if you
>> want me to take care of these while you focus on the main patch? I
>> think the first patch needs to be backpatched till 13 and the second
>> one is for just HEAD.
>>
> 
> Yeah, I plan to push the fixes later today. I'll polish them a bit
> first, and merge the tests (shared by Hou zj) into the patches etc.
> 

I've pushed (and backpatched to 13+) the fix for the publish_as_relid
issue, including the test. I tweaked the test a bit, to check both
orderings of the publication list.

While doing that, I discovered yet ANOTHER bug in the publish_as_relid
loop, affecting 12+13. There was a break once all actions were
replicated, but skipping additional publications ignores the fact that
the publications may replicate a different (higher-up) ancestor.

I removed the break, if anyone thinks this optimization is worth it we
could still do that once we replicate the top-most ancestor.


I'll push the second fix soon.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  180. Re: Column Filtering in Logical Replication

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-03-17T14:17:29Z

    I notice that the publication.sql regression tests contain a number of 
comments like

+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);

but the error doesn't actually happen, because of the way the replica 
identity checking was changed.  This needs to be checked again.

  181. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-17T19:11:32Z

    On 3/17/22 15:17, Peter Eisentraut wrote:
> I notice that the publication.sql regression tests contain a number of
> comments like
> 
> +-- error: replica identity "a" not included in the column list
> +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
> 
> but the error doesn't actually happen, because of the way the replica
> identity checking was changed.  This needs to be checked again.

But the comment describes the error for the whole block, which looks
like this:

-- error: replica identity "a" not included in the column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ERROR:  cannot update table "testpub_tbl5"
DETAIL:  Column list used by the publication does not cover the replica
identity.

So IMHO the comment is correct.

But there was one place where it wasn't entirely clear, as the block was
split by another comment. So I tweaked it to:

-- error: change the replica identity to "b", and column list to (a, c)
-- then update fails, because (a, c) does not cover replica identity

Attached is a rebased patch, on top of the two fixes I pushed.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  182. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-17T19:17:55Z

    I pushed the second fix. Interestingly enough, wrasse failed in the
013_partition test. I don't see how that could be caused by this
particular commit, though - see the pgsql-committers thread [1].

I'd like to test & polish the main patch over the weekend, and get it
committed early next week. Unless someone thinks it's definitely not
ready for that ...


[1]
https://www.postgresql.org/message-id/E1nUsch-0008rQ-FW%40gemulon.postgresql.org

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  183. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-18T05:52:48Z

    On Fri, Mar 18, 2022 at 12:47 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> I pushed the second fix. Interestingly enough, wrasse failed in the
> 013_partition test. I don't see how that could be caused by this
> particular commit, though - see the pgsql-committers thread [1].
>

I have a theory about what's going on here. I think this is due to a
test added in your previous commit c91f71b9dc. The newly added test
added hangs in tablesync because there was no apply worker to set the
state to SUBREL_STATE_CATCHUP which blocked tablesync workers from
proceeding.

See below logs from pogona [1].
2022-03-18 01:33:15.190 CET [2551176][client
backend][3/74:0][013_partition.pl] LOG:  statement: ALTER SUBSCRIPTION
sub2 SET PUBLICATION pub_lower_level, pub_all
2022-03-18 01:33:15.354 CET [2551193][logical replication
worker][4/57:0][] LOG:  logical replication apply worker for
subscription "sub2" has started
2022-03-18 01:33:15.605 CET [2551176][client
backend][:0][013_partition.pl] LOG:  disconnection: session time:
0:00:00.415 user=bf database=postgres host=[local]
2022-03-18 01:33:15.607 CET [2551209][logical replication
worker][3/76:0][] LOG:  logical replication table synchronization
worker for subscription "sub2", table "tab4_1" has started
2022-03-18 01:33:15.609 CET [2551211][logical replication
worker][5/11:0][] LOG:  logical replication table synchronization
worker for subscription "sub2", table "tab3" has started
2022-03-18 01:33:15.617 CET [2551193][logical replication
worker][4/62:0][] LOG:  logical replication apply worker for
subscription "sub2" will restart because of a parameter change

You will notice that the apply worker is never restarted after a
parameter change. The reason was that the particular subscription
reaches the limit of max_sync_workers_per_subscription after which we
don't allow to restart the apply worker. I think you might want to
increase the values of
max_sync_workers_per_subscription/max_logical_replication_workers to
make it work.

> I'd like to test & polish the main patch over the weekend, and get it
> committed early next week. Unless someone thinks it's definitely not
> ready for that ...
>

I think it is in good shape but apart from cleanup, there are issues
with dependency handling which I have analyzed and reported as one of
the comments in the email [2]. I was getting some weird behavior
during my testing due to that. Apart from that still the patch has DDL
handling code in tablecmds.c which probably is not required.
Similarly, Shi-San has reported an issue with replica full in her
email [3]. It is up to you what to do here but it would be good if you
can once share the patch after fixing these issues so that we can
re-test/review it.


[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=pogona&dt=2022-03-17%2023%3A10%3A04
[2] - https://www.postgresql.org/message-id/CAA4eK1KR%2ByUQquK0Bx9uO3eb5xB1e0rAD9xKf-ddm5nSf4WfNg%40mail.gmail.com
[3] - https://www.postgresql.org/message-id/TYAPR01MB6315D664D926EF66DD6E91FCFD109%40TYAPR01MB6315.jpnprd01.prod.outlook.com

-- 
With Regards,
Amit Kapila.

  184. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-18T14:43:41Z

    
On 3/18/22 06:52, Amit Kapila wrote:
> On Fri, Mar 18, 2022 at 12:47 AM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> I pushed the second fix. Interestingly enough, wrasse failed in the
>> 013_partition test. I don't see how that could be caused by this
>> particular commit, though - see the pgsql-committers thread [1].
>>
> 
> I have a theory about what's going on here. I think this is due to a
> test added in your previous commit c91f71b9dc. The newly added test
> added hangs in tablesync because there was no apply worker to set the
> state to SUBREL_STATE_CATCHUP which blocked tablesync workers from
> proceeding.
> 
> See below logs from pogona [1].
> 2022-03-18 01:33:15.190 CET [2551176][client
> backend][3/74:0][013_partition.pl] LOG:  statement: ALTER SUBSCRIPTION
> sub2 SET PUBLICATION pub_lower_level, pub_all
> 2022-03-18 01:33:15.354 CET [2551193][logical replication
> worker][4/57:0][] LOG:  logical replication apply worker for
> subscription "sub2" has started
> 2022-03-18 01:33:15.605 CET [2551176][client
> backend][:0][013_partition.pl] LOG:  disconnection: session time:
> 0:00:00.415 user=bf database=postgres host=[local]
> 2022-03-18 01:33:15.607 CET [2551209][logical replication
> worker][3/76:0][] LOG:  logical replication table synchronization
> worker for subscription "sub2", table "tab4_1" has started
> 2022-03-18 01:33:15.609 CET [2551211][logical replication
> worker][5/11:0][] LOG:  logical replication table synchronization
> worker for subscription "sub2", table "tab3" has started
> 2022-03-18 01:33:15.617 CET [2551193][logical replication
> worker][4/62:0][] LOG:  logical replication apply worker for
> subscription "sub2" will restart because of a parameter change
> 
> You will notice that the apply worker is never restarted after a
> parameter change. The reason was that the particular subscription
> reaches the limit of max_sync_workers_per_subscription after which we
> don't allow to restart the apply worker. I think you might want to
> increase the values of
> max_sync_workers_per_subscription/max_logical_replication_workers to
> make it work.
> 

Hmmm. So the theory is that in most runs we manage to sync the tables
faster than starting the workers, so we don't hit the limit. But on some
machines the sync worker takes a bit longer, we hit the limit. Seems
possible, yes. Unfortunately we don't seem to log anything when we hit
the limit, so hard to say for sure :-( I suggest we add a WARNING
message to logicalrep_worker_launch or something. Not just because of
this test, it seems useful in general.

However, how come we don't retry the sync? Surely we don't just give up
forever, that'd be a pretty annoying behavior. Presumably we just end up
sleeping for a long time before restarting the sync worker, somewhere.

>> I'd like to test & polish the main patch over the weekend, and get it
>> committed early next week. Unless someone thinks it's definitely not
>> ready for that ...
>>
> 
> I think it is in good shape but apart from cleanup, there are issues
> with dependency handling which I have analyzed and reported as one of
> the comments in the email [2]. I was getting some weird behavior
> during my testing due to that. Apart from that still the patch has DDL
> handling code in tablecmds.c which probably is not required.
> Similarly, Shi-San has reported an issue with replica full in her
> email [3]. It is up to you what to do here but it would be good if you
> can once share the patch after fixing these issues so that we can
> re-test/review it.

Ah, thanks for reminding me - it's hard to keep track of all the issues
in threads as long as this one.

BTW do you have any opinion on the SET COLUMNS syntax? Peter Smith
proposed to get rid of it in [1] but I'm not sure that's a good idea.
Because if we ditch it, then removing the column list would look like this:

    ALTER PUBLICATION pub ALTER TABLE tab;

And if we happen to add other per-table options, this would become
pretty ambiguous.

Actually, do we even want to allow resetting column lists like this? We
don't allow this for row filters, so if you want to change a row filter
you have to re-add the table, right? So maybe we should just ditch ALTER
TABLE entirely.

regards

[4]
https://www.postgresql.org/message-id/CAHut%2BPtc7Rh187eQKrxdUmUNWyfxz7OkhYAX%3DAW411Qwxya0LQ%40mail.gmail.com

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  185. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-18T17:12:20Z

    
On 3/18/22 15:43, Tomas Vondra wrote:
> 
> 
> On 3/18/22 06:52, Amit Kapila wrote:
>> On Fri, Mar 18, 2022 at 12:47 AM Tomas Vondra
>> <tomas.vondra@enterprisedb.com> wrote:
>>>
>>> I pushed the second fix. Interestingly enough, wrasse failed in the
>>> 013_partition test. I don't see how that could be caused by this
>>> particular commit, though - see the pgsql-committers thread [1].
>>>
>>
>> I have a theory about what's going on here. I think this is due to a
>> test added in your previous commit c91f71b9dc. The newly added test
>> added hangs in tablesync because there was no apply worker to set the
>> state to SUBREL_STATE_CATCHUP which blocked tablesync workers from
>> proceeding.
>>
>> See below logs from pogona [1].
>> 2022-03-18 01:33:15.190 CET [2551176][client
>> backend][3/74:0][013_partition.pl] LOG:  statement: ALTER SUBSCRIPTION
>> sub2 SET PUBLICATION pub_lower_level, pub_all
>> 2022-03-18 01:33:15.354 CET [2551193][logical replication
>> worker][4/57:0][] LOG:  logical replication apply worker for
>> subscription "sub2" has started
>> 2022-03-18 01:33:15.605 CET [2551176][client
>> backend][:0][013_partition.pl] LOG:  disconnection: session time:
>> 0:00:00.415 user=bf database=postgres host=[local]
>> 2022-03-18 01:33:15.607 CET [2551209][logical replication
>> worker][3/76:0][] LOG:  logical replication table synchronization
>> worker for subscription "sub2", table "tab4_1" has started
>> 2022-03-18 01:33:15.609 CET [2551211][logical replication
>> worker][5/11:0][] LOG:  logical replication table synchronization
>> worker for subscription "sub2", table "tab3" has started
>> 2022-03-18 01:33:15.617 CET [2551193][logical replication
>> worker][4/62:0][] LOG:  logical replication apply worker for
>> subscription "sub2" will restart because of a parameter change
>>
>> You will notice that the apply worker is never restarted after a
>> parameter change. The reason was that the particular subscription
>> reaches the limit of max_sync_workers_per_subscription after which we
>> don't allow to restart the apply worker. I think you might want to
>> increase the values of
>> max_sync_workers_per_subscription/max_logical_replication_workers to
>> make it work.
>>
> 
> Hmmm. So the theory is that in most runs we manage to sync the tables
> faster than starting the workers, so we don't hit the limit. But on some
> machines the sync worker takes a bit longer, we hit the limit. Seems
> possible, yes. Unfortunately we don't seem to log anything when we hit
> the limit, so hard to say for sure :-( I suggest we add a WARNING
> message to logicalrep_worker_launch or something. Not just because of
> this test, it seems useful in general.
> 
> However, how come we don't retry the sync? Surely we don't just give up
> forever, that'd be a pretty annoying behavior. Presumably we just end up
> sleeping for a long time before restarting the sync worker, somewhere.
> 

I tried lowering the max_sync_workers_per_subscription to 1 and making
the workers to run for a couple seconds (doing some CPU intensive
stuff), but everything still works just fine.

Looking a bit closer at the logs (from pogona and other), I doubt this
is about hitting the max_sync_workers_per_subscription limit. Notice we
start two sync workers, but neither of them ever completes. So we never
update the sync status or start syncing the remaining tables.

So the question is why those two sync workers never complete - I guess
there's some sort of lock wait (deadlock?) or infinite loop.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  186. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-18T22:26:39Z

    
On 3/18/22 15:43, Tomas Vondra wrote:
> 
> 
> On 3/18/22 06:52, Amit Kapila wrote:
>>
>>  ...
>> I think it is in good shape but apart from cleanup, there are issues
>> with dependency handling which I have analyzed and reported as one of
>> the comments in the email [2]. I was getting some weird behavior
>> during my testing due to that. Apart from that still the patch has DDL
>> handling code in tablecmds.c which probably is not required.
>> Similarly, Shi-San has reported an issue with replica full in her
>> email [3]. It is up to you what to do here but it would be good if you
>> can once share the patch after fixing these issues so that we can
>> re-test/review it.
> 
> Ah, thanks for reminding me - it's hard to keep track of all the issues
> in threads as long as this one.
> 

Attached is an updated patch, hopefully addressing these issues.

Firstly, I've reverted the changes in tablecmds.c, instead relying on
regular dependency behavior. I've also switched from DEPENDENCY_AUTO to
DEPENDENCY_NORMAL. This makes the code simpler, and the behavior should
be the same as for row filters, which makes it more consistent.

As for the SET COLUMNS breaking behaviors, I've decided to drop this
feature entirely, for the reasons outlined earlier today. We don't have
that for row filters either, etc. This means the dependency issue simply
disappears.

Without SET COLUMNS, if you want to change the column list you have to
remove the table from the subscription, and add it back (with the new
column list). Perhaps inconvenient, but the behavior is clearly defined.
Maybe we need a more convenient way to tweak column lists, but I'd say
we should have the same thing for row filters too.


As for the issue reported by Shi-San about replica identity full and
column filters, presumably you're referring to this:

  create table tbl (a int, b int, c int);
  create publication pub for table tbl (a, b, c);
  alter table tbl replica identity full;

  postgres=# delete from tbl;
  ERROR:  cannot delete from table "tbl"
  DETAIL:  Column list used by the publication does not cover the
           replica identity.

I believe not allowing column lists with REPLICA IDENTITY FULL is
expected / correct behavior. I mean, for that to work the column list
has to always include all columns anyway, so it's pretty pointless. Of
course, we might check that the column list contains everything, but
considering the list does always have to contain all columns, and it
break as soon as you add any columns, it seems reasonable (cheaper) to
just require no column lists.

I also went through the patch and made the naming more consistent. The
comments used both "column filter" and "column list" randomly, and I
think the agreement is to use "list" so I adopted that wording.


However, while looking at how pgoutput, I realized one thing - for row
filters we track them "per operation", depending on which operations are
defined for a given publication. Shouldn't we do the same thing for
column lists, really?

I mean, if there are two publications with different column lists, one
for inserts and the other one for updates, isn't it wrong to merge these
two column lists?

Also, doesn't this mean publish_as_relid should be "per operation" too?


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  187. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-19T17:11:38Z

    Fix a compiler warning reported by cfbot.

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  188. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-19T17:41:15Z

    On 3/19/22 18:11, Tomas Vondra wrote:
> Fix a compiler warning reported by cfbot.

Apologies, I failed to actually commit the fix. So here we go again.

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  189. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-20T03:11:40Z

    On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/18/22 15:43, Tomas Vondra wrote:
> >>
> >
> > Hmmm. So the theory is that in most runs we manage to sync the tables
> > faster than starting the workers, so we don't hit the limit. But on some
> > machines the sync worker takes a bit longer, we hit the limit. Seems
> > possible, yes. Unfortunately we don't seem to log anything when we hit
> > the limit, so hard to say for sure :-( I suggest we add a WARNING
> > message to logicalrep_worker_launch or something. Not just because of
> > this test, it seems useful in general.
> >
> > However, how come we don't retry the sync? Surely we don't just give up
> > forever, that'd be a pretty annoying behavior. Presumably we just end up
> > sleeping for a long time before restarting the sync worker, somewhere.
> >
>
> I tried lowering the max_sync_workers_per_subscription to 1 and making
> the workers to run for a couple seconds (doing some CPU intensive
> stuff), but everything still works just fine.
>

Did the apply worker restarts during that time? If not you can try by
changing some subscription parameters which leads to its restart. This
has to happen before copy_table has finished. In the LOGS, you should
see the message: "logical replication apply worker for subscription
"<subscription_name>" will restart because of a parameter change".
IIUC, the code which doesn't allow to restart the apply worker after
the max_sync_workers_per_subscription is reached is as below:
logicalrep_worker_launch()
{
...
if (nsyncworkers >= max_sync_workers_per_subscription)
{
LWLockRelease(LogicalRepWorkerLock);
return;
}
...
}

This happens before we allocate a worker to apply. So, it can happen
only during the restart of the apply worker because we always first
the apply worker, so in that case, it will never restart.

> Looking a bit closer at the logs (from pogona and other), I doubt this
> is about hitting the max_sync_workers_per_subscription limit. Notice we
> start two sync workers, but neither of them ever completes. So we never
> update the sync status or start syncing the remaining tables.
>

I think they are never completed because they are in a sort of
infinite loop. If you see process_syncing_tables_for_sync(), it will
never mark the status as SUBREL_STATE_SYNCDONE unless apply worker has
set it to SUBREL_STATE_CATCHUP. In LogicalRepSyncTableStart(), we do
wait for a state change to catchup via wait_for_worker_state_change(),
but we bail out in that function if the apply worker has died. After
that tablesync worker won't be able to complete because in our case
apply worker won't be able to restart.

> So the question is why those two sync workers never complete - I guess
> there's some sort of lock wait (deadlock?) or infinite loop.
>

It would be a bit tricky to reproduce this even if the above theory is
correct but I'll try it today or tomorrow.

-- 
With Regards,
Amit Kapila.

  190. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-20T06:23:41Z

    On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>
> > So the question is why those two sync workers never complete - I guess
> > there's some sort of lock wait (deadlock?) or infinite loop.
> >
>
> It would be a bit tricky to reproduce this even if the above theory is
> correct but I'll try it today or tomorrow.
>

I am able to reproduce it with the help of a debugger. Firstly, I have
added the LOG message and some While (true) loops to debug sync and
apply workers. Test setup

Node-1:
create table t1(c1);
create table t2(c1);
insert into t1 values(1);
create publication pub1 for table t1;
create publication pu2;

Node-2:
change max_sync_workers_per_subscription to 1 in potgresql.conf
create table t1(c1);
create table t2(c1);
create subscription sub1 connection 'dbname = postgres' publication pub1;

Till this point, just allow debuggers in both workers just continue.

Node-1:
alter publication pub1 add table t2;
insert into t1 values(2);

Here, we have to debug the apply worker such that when it tries to
apply the insert, stop the debugger in function apply_handle_insert()
after doing begin_replication_step().

Node-2:
alter subscription sub1 set pub1, pub2;

Now, continue the debugger of apply worker, it should first start the
sync worker and then exit because of parameter change. All of these
debugging steps are to just ensure the point that it should first
start the sync worker and then exit. After this point, table sync
worker never finishes and log is filled with messages: "reached
max_sync_workers_per_subscription limit" (a newly added message by me
in the attached debug patch).

Now, it is not completely clear to me how exactly '013_partition.pl'
leads to this situation but there is a possibility based on the LOGs
it shows.

-- 
With Regards,
Amit Kapila.

  191. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-20T11:23:39Z

    
On 3/20/22 07:23, Amit Kapila wrote:
> On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
>> <tomas.vondra@enterprisedb.com> wrote:
>>
>>> So the question is why those two sync workers never complete - I guess
>>> there's some sort of lock wait (deadlock?) or infinite loop.
>>>
>>
>> It would be a bit tricky to reproduce this even if the above theory is
>> correct but I'll try it today or tomorrow.
>>
> 
> I am able to reproduce it with the help of a debugger. Firstly, I have
> added the LOG message and some While (true) loops to debug sync and
> apply workers. Test setup
> 
> Node-1:
> create table t1(c1);
> create table t2(c1);
> insert into t1 values(1);
> create publication pub1 for table t1;
> create publication pu2;
> 
> Node-2:
> change max_sync_workers_per_subscription to 1 in potgresql.conf
> create table t1(c1);
> create table t2(c1);
> create subscription sub1 connection 'dbname = postgres' publication pub1;
> 
> Till this point, just allow debuggers in both workers just continue.
> 
> Node-1:
> alter publication pub1 add table t2;
> insert into t1 values(2);
> 
> Here, we have to debug the apply worker such that when it tries to
> apply the insert, stop the debugger in function apply_handle_insert()
> after doing begin_replication_step().
> 
> Node-2:
> alter subscription sub1 set pub1, pub2;
> 
> Now, continue the debugger of apply worker, it should first start the
> sync worker and then exit because of parameter change. All of these
> debugging steps are to just ensure the point that it should first
> start the sync worker and then exit. After this point, table sync
> worker never finishes and log is filled with messages: "reached
> max_sync_workers_per_subscription limit" (a newly added message by me
> in the attached debug patch).
> 
> Now, it is not completely clear to me how exactly '013_partition.pl'
> leads to this situation but there is a possibility based on the LOGs
> it shows.
> 

Thanks, I'll take a look later. From the description it seems this is an
issue that existed before any of the patches, right? It might be more
likely to hit due to some test changes, but the root cause is older.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  192. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-21T10:38:59Z

    On Sun, Mar 20, 2022 at 4:53 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/20/22 07:23, Amit Kapila wrote:
> > On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >>
> >> On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
> >> <tomas.vondra@enterprisedb.com> wrote:
> >>
> >>> So the question is why those two sync workers never complete - I guess
> >>> there's some sort of lock wait (deadlock?) or infinite loop.
> >>>
> >>
> >> It would be a bit tricky to reproduce this even if the above theory is
> >> correct but I'll try it today or tomorrow.
> >>
> >
> > I am able to reproduce it with the help of a debugger. Firstly, I have
> > added the LOG message and some While (true) loops to debug sync and
> > apply workers. Test setup
> >
> > Node-1:
> > create table t1(c1);
> > create table t2(c1);
> > insert into t1 values(1);
> > create publication pub1 for table t1;
> > create publication pu2;
> >
> > Node-2:
> > change max_sync_workers_per_subscription to 1 in potgresql.conf
> > create table t1(c1);
> > create table t2(c1);
> > create subscription sub1 connection 'dbname = postgres' publication pub1;
> >
> > Till this point, just allow debuggers in both workers just continue.
> >
> > Node-1:
> > alter publication pub1 add table t2;
> > insert into t1 values(2);
> >
> > Here, we have to debug the apply worker such that when it tries to
> > apply the insert, stop the debugger in function apply_handle_insert()
> > after doing begin_replication_step().
> >
> > Node-2:
> > alter subscription sub1 set pub1, pub2;
> >
> > Now, continue the debugger of apply worker, it should first start the
> > sync worker and then exit because of parameter change. All of these
> > debugging steps are to just ensure the point that it should first
> > start the sync worker and then exit. After this point, table sync
> > worker never finishes and log is filled with messages: "reached
> > max_sync_workers_per_subscription limit" (a newly added message by me
> > in the attached debug patch).
> >
> > Now, it is not completely clear to me how exactly '013_partition.pl'
> > leads to this situation but there is a possibility based on the LOGs
> > it shows.
> >
>
> Thanks, I'll take a look later. From the description it seems this is an
> issue that existed before any of the patches, right? It might be more
> likely to hit due to some test changes, but the root cause is older.
>

Yes, your understanding is correct. If my understanding is correct,
then we need probably just need some changes in the new test to make
it behave as per the current code.

-- 
With Regards,
Amit Kapila.

  193. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-21T11:28:53Z

    On Fri, Mar 18, 2022 at 8:13 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> Ah, thanks for reminding me - it's hard to keep track of all the issues
> in threads as long as this one.
>
> BTW do you have any opinion on the SET COLUMNS syntax? Peter Smith
> proposed to get rid of it in [1] but I'm not sure that's a good idea.
> Because if we ditch it, then removing the column list would look like this:
>
>     ALTER PUBLICATION pub ALTER TABLE tab;
>
> And if we happen to add other per-table options, this would become
> pretty ambiguous.
>
> Actually, do we even want to allow resetting column lists like this? We
> don't allow this for row filters, so if you want to change a row filter
> you have to re-add the table, right?
>

We can use syntax like: "alter publication pub1 set table t1 where (c2
> 10);" to reset the existing row filter. It seems similar thing works
for column list as well ("alter publication pub1 set table t1 (c2)
where (c2 > 10)"). If I am not missing anything, I don't think we need
additional Alter Table syntax.

> So maybe we should just ditch ALTER
> TABLE entirely.
>

Yeah, I agree especially if my above understanding is correct.

-- 
With Regards,
Amit Kapila.

  194. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-21T11:55:28Z

    On Sat, Mar 19, 2022 at 3:56 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/18/22 15:43, Tomas Vondra wrote:
> >
>
> As for the issue reported by Shi-San about replica identity full and
> column filters, presumably you're referring to this:
>
>   create table tbl (a int, b int, c int);
>   create publication pub for table tbl (a, b, c);
>   alter table tbl replica identity full;
>
>   postgres=# delete from tbl;
>   ERROR:  cannot delete from table "tbl"
>   DETAIL:  Column list used by the publication does not cover the
>            replica identity.
>
> I believe not allowing column lists with REPLICA IDENTITY FULL is
> expected / correct behavior. I mean, for that to work the column list
> has to always include all columns anyway, so it's pretty pointless. Of
> course, we might check that the column list contains everything, but
> considering the list does always have to contain all columns, and it
> break as soon as you add any columns, it seems reasonable (cheaper) to
> just require no column lists.
>

Fair point. We can leave this as it is.

> I also went through the patch and made the naming more consistent. The
> comments used both "column filter" and "column list" randomly, and I
> think the agreement is to use "list" so I adopted that wording.
>
>
> However, while looking at how pgoutput, I realized one thing - for row
> filters we track them "per operation", depending on which operations are
> defined for a given publication. Shouldn't we do the same thing for
> column lists, really?
>
> I mean, if there are two publications with different column lists, one
> for inserts and the other one for updates, isn't it wrong to merge these
> two column lists?
>

The reason we can't combine row filters for inserts with
updates/deletes is that if inserts have some column that is not
present in RI then during update filtering (for old tuple) it will
give an error as the column won't be present in WAL log.

OTOH, the same problem won't be there for the column list/filter patch
because all the RI columns are there in the column list (for
update/delete) and we don't need to apply a column filter for old
tuples in either update or delete.

Basically, the filter rules are slightly different for row filters and
column lists, so we need them (combine of filters) for one but not for
the other. Now, for the sake of consistency with row filters, we can
do it but as such there won't be any problem or maybe we can just add
a comment for the same in code.

-- 
With Regards,
Amit Kapila.

  195. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-03-21T12:16:37Z

    Hello,

Please add me to the list of authors of this patch.  I made a large
number of nontrivial changes to it early on.  Thanks.  I have modified
the entry in the CF app (which sorts alphabetically, it was not my
intention to put my name first.)

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/

  196. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-21T14:12:04Z

    On Sat, Mar 19, 2022 at 11:11 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/19/22 18:11, Tomas Vondra wrote:
> > Fix a compiler warning reported by cfbot.
>
> Apologies, I failed to actually commit the fix. So here we go again.
>

Few comments:
===============
1.
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
{
...
}

...
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
{
...
}

Both these functions are not required now. So, we can remove them.

2.
@@ -464,11 +478,11 @@ logicalrep_write_update(StringInfo out,
TransactionId xid, Relation rel,
  pq_sendbyte(out, 'O'); /* old tuple follows */
  else
  pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
  }

As mentioned previously, here, we should pass NULL similar to
logicalrep_write_delete as we don't need to use column list for old
tuples.

3.
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+    bool pubviaroot)

The second parameter is not used in this function. As noted in the
comments, I also think it is better to rename this. How about
ValidatePubColumnList?

4.
@@ -821,6 +942,9 @@ fetch_remote_table_info(char *nspname, char *relname,
  *
  * 3) one of the subscribed publications is declared as ALL TABLES IN
  * SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
  */
  if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)

Why is this comment added in the row filter code? Now, both row filter
and column list are fetched in the same way, so not sure what exactly
this comment is referring to.

5.
+/* qsort comparator for attnums */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}

The exact same code exists in statscmds.c. Do we need a second copy of the same?

6.
 static void pgoutput_row_filter_init(PGOutputData *data,
  List *publications,
  RelationSyncEntry *entry);
+
 static bool pgoutput_row_filter_exec_expr(ExprState *state,

Spurious line addition.

7. The tests in 030_column_list.pl take a long time as compared to all
other similar individual tests in the subscription folder. I haven't
checked whether there is any need to reduce some tests but it seems
worth checking.

-- 
With Regards,
Amit Kapila.

  197. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-03-22T19:24:57Z

    On 2022-Mar-19, Tomas Vondra wrote:

> @@ -174,7 +182,13 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
>    <para>
>     Add some tables to the publication:
>  <programlisting>
> -ALTER PUBLICATION mypublication ADD TABLE users, departments;
> +ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
> +</programlisting></para>
> +
> +  <para>
> +   Change the set of columns published for a table:
> +<programlisting>
> +ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname, lastname), TABLE departments;
>  </programlisting></para>
>  
>    <para>

Hmm, it seems to me that if you've removed the feature to change the set
of columns published for a table, then the second example should be
removed as well.

> +/*
> + * Transform the publication column lists expression for all the relations
> + * in the list.
> + *
> + * XXX The name is a bit misleading, because we don't really transform
> + * anything here - we merely check the column list is compatible with the
> + * definition of the publication (with publish_via_partition_root=false)
> + * we only allow column lists on the leaf relations. So maybe rename it?
> + */
> +static void
> +TransformPubColumnList(List *tables, const char *queryString,
> +					   bool pubviaroot)
> +{

I agree with renaming this function.  Maybe CheckPubRelationColumnList() ?

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"This is a foot just waiting to be shot"                (Andrew Dunstan)

  198. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-23T02:17:21Z

    On Wed, Mar 23, 2022 at 12:54 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2022-Mar-19, Tomas Vondra wrote:
>
> > @@ -174,7 +182,13 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
> >    <para>
> >     Add some tables to the publication:
> >  <programlisting>
> > -ALTER PUBLICATION mypublication ADD TABLE users, departments;
> > +ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
> > +</programlisting></para>
> > +
> > +  <para>
> > +   Change the set of columns published for a table:
> > +<programlisting>
> > +ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname, lastname), TABLE departments;
> >  </programlisting></para>
> >
> >    <para>
>
> Hmm, it seems to me that if you've removed the feature to change the set
> of columns published for a table, then the second example should be
> removed as well.
>

As per my understanding, the removed feature is "Alter Publication ...
Alter Table ...". The example here "Alter Publication ... Set Table
.." should still work as mentioned in my email[1].

[1] - https://www.postgresql.org/message-id/CAA4eK1L6YTcx%3DyJfdudr-y98Wcn4rWX4puHGAa2nxSCRb3fzQw%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  199. Re: Column Filtering in Logical Replication

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-03-23T10:21:23Z

    On 2022-Mar-23, Amit Kapila wrote:

> On Wed, Mar 23, 2022 at 12:54 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> >
> > On 2022-Mar-19, Tomas Vondra wrote:
> >
> > > @@ -174,7 +182,13 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
> > >    <para>
> > >     Add some tables to the publication:
> > >  <programlisting>
> > > -ALTER PUBLICATION mypublication ADD TABLE users, departments;
> > > +ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
> > > +</programlisting></para>
> > > +
> > > +  <para>
> > > +   Change the set of columns published for a table:
> > > +<programlisting>
> > > +ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname, lastname), TABLE departments;
> > >  </programlisting></para>
> > >
> > >    <para>
> >
> > Hmm, it seems to me that if you've removed the feature to change the set
> > of columns published for a table, then the second example should be
> > removed as well.
> 
> As per my understanding, the removed feature is "Alter Publication ...
> Alter Table ...". The example here "Alter Publication ... Set Table
> .." should still work as mentioned in my email[1].

Ah, I see.  Yeah, that makes sense.  In that case, the leading text
seems a bit confusing.  I would suggest "Change the set of tables in the
publication, specifying a different set of columns for one of them:"

I think it would make the example more useful if we table for which the
columns are changing is a different one.  Maybe do this:

    Add some tables to the publication:
 <programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+  <para>
+   Change the set of tables in the publication, keeping the column list
+   in the users table and specifying a different column list for the
+   departments table. Note that previously published tables not mentioned
+   in this command are removed from the publication:
+
+<programlisting>
+ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname), TABLE departments (dept_id, deptname);
 </programlisting></para>

so that it is clear that if you want to keep the column list unchanged
in one table, you are forced to specify it again.

(Frankly, this ALTER PUBLICATION SET command seems pretty useless from a
user PoV.)

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Investigación es lo que hago cuando no sé lo que estoy haciendo"
(Wernher von Braun)

  200. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-23T22:37:54Z

    On 3/21/22 12:55, Amit Kapila wrote:
> On Sat, Mar 19, 2022 at 3:56 AM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> ...
>>
>> However, while looking at how pgoutput, I realized one thing - for row
>> filters we track them "per operation", depending on which operations are
>> defined for a given publication. Shouldn't we do the same thing for
>> column lists, really?
>>
>> I mean, if there are two publications with different column lists, one
>> for inserts and the other one for updates, isn't it wrong to merge these
>> two column lists?
>>
> 
> The reason we can't combine row filters for inserts with
> updates/deletes is that if inserts have some column that is not
> present in RI then during update filtering (for old tuple) it will
> give an error as the column won't be present in WAL log.
> 
> OTOH, the same problem won't be there for the column list/filter patch
> because all the RI columns are there in the column list (for
> update/delete) and we don't need to apply a column filter for old
> tuples in either update or delete.
> 
> Basically, the filter rules are slightly different for row filters and
> column lists, so we need them (combine of filters) for one but not for
> the other. Now, for the sake of consistency with row filters, we can
> do it but as such there won't be any problem or maybe we can just add
> a comment for the same in code.
> 

OK, thanks for the explanation. I'll add a comment explaining this to
the function initializing the column filter.

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  201. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-23T22:41:48Z

    
On 3/21/22 12:28, Amit Kapila wrote:
> On Fri, Mar 18, 2022 at 8:13 PM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> Ah, thanks for reminding me - it's hard to keep track of all the issues
>> in threads as long as this one.
>>
>> BTW do you have any opinion on the SET COLUMNS syntax? Peter Smith
>> proposed to get rid of it in [1] but I'm not sure that's a good idea.
>> Because if we ditch it, then removing the column list would look like this:
>>
>>     ALTER PUBLICATION pub ALTER TABLE tab;
>>
>> And if we happen to add other per-table options, this would become
>> pretty ambiguous.
>>
>> Actually, do we even want to allow resetting column lists like this? We
>> don't allow this for row filters, so if you want to change a row filter
>> you have to re-add the table, right?
>>
> 
> We can use syntax like: "alter publication pub1 set table t1 where (c2
>> 10);" to reset the existing row filter. It seems similar thing works
> for column list as well ("alter publication pub1 set table t1 (c2)
> where (c2 > 10)"). If I am not missing anything, I don't think we need
> additional Alter Table syntax.
> 
>> So maybe we should just ditch ALTER
>> TABLE entirely.
>>
> 
> Yeah, I agree especially if my above understanding is correct.
> 

I think there's a gotcha that

   ALTER PUBLICATION pub SET TABLE t ...

also removes all other relations from the publication, and it removes
and re-adds the table anyway. So I'm not sure what's the advantage?

Anyway, I don't see why we would need such ALTER TABLE only for column
filters and not for row filters - either we need to allow this for both
options or none of them.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  202. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-24T02:58:25Z

    On Thu, Mar 24, 2022 at 4:11 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/21/22 12:28, Amit Kapila wrote:
> > On Fri, Mar 18, 2022 at 8:13 PM Tomas Vondra
> > <tomas.vondra@enterprisedb.com> wrote:
> >>
> >> Ah, thanks for reminding me - it's hard to keep track of all the issues
> >> in threads as long as this one.
> >>
> >> BTW do you have any opinion on the SET COLUMNS syntax? Peter Smith
> >> proposed to get rid of it in [1] but I'm not sure that's a good idea.
> >> Because if we ditch it, then removing the column list would look like this:
> >>
> >>     ALTER PUBLICATION pub ALTER TABLE tab;
> >>
> >> And if we happen to add other per-table options, this would become
> >> pretty ambiguous.
> >>
> >> Actually, do we even want to allow resetting column lists like this? We
> >> don't allow this for row filters, so if you want to change a row filter
> >> you have to re-add the table, right?
> >>
> >
> > We can use syntax like: "alter publication pub1 set table t1 where (c2
> >> 10);" to reset the existing row filter. It seems similar thing works
> > for column list as well ("alter publication pub1 set table t1 (c2)
> > where (c2 > 10)"). If I am not missing anything, I don't think we need
> > additional Alter Table syntax.
> >
> >> So maybe we should just ditch ALTER
> >> TABLE entirely.
> >>
> >
> > Yeah, I agree especially if my above understanding is correct.
> >
>
> I think there's a gotcha that
>
>    ALTER PUBLICATION pub SET TABLE t ...
>
> also removes all other relations from the publication, and it removes
> and re-adds the table anyway. So I'm not sure what's the advantage?
>

I think it could be used when the user has fewer tables and she wants
to change the list of published tables or their row/column filters. I
am not sure of the value of this to users but this was a pre-existing
syntax.

> Anyway, I don't see why we would need such ALTER TABLE only for column
> filters and not for row filters - either we need to allow this for both
> options or none of them.
>

+1. I think for now we can leave this new ALTER TABLE syntax and do it
for both column and row filters together.


-- 
With Regards,
Amit Kapila.

  203. Re: Column Filtering in Logical Replication

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-03-24T16:33:56Z

    On 17.03.22 20:11, Tomas Vondra wrote:
> But the comment describes the error for the whole block, which looks
> like this:
> 
> -- error: replica identity "a" not included in the column list
> ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
> UPDATE testpub_tbl5 SET a = 1;
> ERROR:  cannot update table "testpub_tbl5"
> DETAIL:  Column list used by the publication does not cover the replica
> identity.
> 
> So IMHO the comment is correct.

Ok, that makes sense.  I read all the comments in the test file again. 
There were a couple that I think could use tweaking; see attached file. 
The ones with "???" didn't make sense to me:  The first one is before a 
command that doesn't seem to change anything, the second one I didn't 
understand the meaning.  Please take a look.

(The patch is actually based on your 20220318c patch, but I'm adding it 
here since we have the discussion here.)

  204. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-25T00:14:48Z

    
On 3/24/22 17:33, Peter Eisentraut wrote:
> 
> On 17.03.22 20:11, Tomas Vondra wrote:
>> But the comment describes the error for the whole block, which looks
>> like this:
>>
>> -- error: replica identity "a" not included in the column list
>> ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
>> UPDATE testpub_tbl5 SET a = 1;
>> ERROR:  cannot update table "testpub_tbl5"
>> DETAIL:  Column list used by the publication does not cover the replica
>> identity.
>>
>> So IMHO the comment is correct.
> 
> Ok, that makes sense.  I read all the comments in the test file again.
> There were a couple that I think could use tweaking; see attached file.
> The ones with "???" didn't make sense to me:  The first one is before a
> command that doesn't seem to change anything, the second one I didn't
> understand the meaning.  Please take a look.
> 

Thanks, the proposed changes seem reasonable. As for the two unclear
tests/comments:

  -- make sure changing the column list is updated in SET TABLE (???)
  CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
  ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
  \d+ testpub_tbl7

  -- ok: we'll skip this table (???)
  ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
  \d+ testpub_tbl7

  -- ok: update the column list
  ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
  \d+ testpub_tbl7

The goal of this test is to verify that we handle column lists correctly
in SET TABLE. That is, if the column list matches the currently set one,
we should just skip the table in SET TABLE. If it's different, we need
to update the catalog. That's what the first comment is trying to say.

It's true we can't really check we skip the table in the SetObject code,
but we can at least ensure there's no error and the column list remains
the same.

And we're not replicating any data in regression tests, so it might
happen we discard the new column list, for example. Hence the second
test, which ensures we end up with the modified column list.

Attached is a patch, rebased on top of the sequence decoding stuff I
pushed earlier today, also including the comments rewording, and
renaming the "transform" function.

I'll go over it again and get it pushed soon, unless someone objects.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  205. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-25T03:10:04Z

    On Fri, Mar 25, 2022 at 5:44 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> Attached is a patch, rebased on top of the sequence decoding stuff I
> pushed earlier today, also including the comments rewording, and
> renaming the "transform" function.
>
> I'll go over it again and get it pushed soon, unless someone objects.
>

You haven't addressed the comments given by me earlier this week. See
https://www.postgresql.org/message-id/CAA4eK1LY_JGL7LvdT64ujEiEAVaADuhdej1QNnwxvO_-KPzeEg%40mail.gmail.com.

*
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+CheckPubRelationColumnList(List *tables, const char *queryString,
+    bool pubviaroot)

After changing this function name, the comment above is not required.

-- 
With Regards,
Amit Kapila.

  206. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-26T00:18:08Z

    
On 3/25/22 04:10, Amit Kapila wrote:
> On Fri, Mar 25, 2022 at 5:44 AM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> Attached is a patch, rebased on top of the sequence decoding stuff I
>> pushed earlier today, also including the comments rewording, and
>> renaming the "transform" function.
>>
>> I'll go over it again and get it pushed soon, unless someone objects.
>>
> 
> You haven't addressed the comments given by me earlier this week. See
> https://www.postgresql.org/message-id/CAA4eK1LY_JGL7LvdT64ujEiEAVaADuhdej1QNnwxvO_-KPzeEg%40mail.gmail.com.
> 

Thanks for noticing that! Thunderbird did not include that message into
the patch thread for some reason, so I did not notice that!

> *
> + * XXX The name is a bit misleading, because we don't really transform
> + * anything here - we merely check the column list is compatible with the
> + * definition of the publication (with publish_via_partition_root=false)
> + * we only allow column lists on the leaf relations. So maybe rename it?
> + */
> +static void
> +CheckPubRelationColumnList(List *tables, const char *queryString,
> +    bool pubviaroot)
> 
> After changing this function name, the comment above is not required.
> 

Thanks, comment updated.

I went over the patch again, polished the commit message a bit, and
pushed. May the buildfarm be merciful!

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  207. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-26T00:29:35Z

    On 3/21/22 15:12, Amit Kapila wrote:
> On Sat, Mar 19, 2022 at 11:11 PM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> On 3/19/22 18:11, Tomas Vondra wrote:
>>> Fix a compiler warning reported by cfbot.
>>
>> Apologies, I failed to actually commit the fix. So here we go again.
>>
> 
> Few comments:
> ===============
> 1.
> +/*
> + * Gets a list of OIDs of all partial-column publications of the given
> + * relation, that is, those that specify a column list.
> + */
> +List *
> +GetRelationColumnPartialPublications(Oid relid)
> {
> ...
> }
> 
> ...
> +/*
> + * For a relation in a publication that is known to have a non-null column
> + * list, return the list of attribute numbers that are in it.
> + */
> +List *
> +GetRelationColumnListInPublication(Oid relid, Oid pubid)
> {
> ...
> }
> 
> Both these functions are not required now. So, we can remove them.
> 

Good catch, removed.

> 2.
> @@ -464,11 +478,11 @@ logicalrep_write_update(StringInfo out,
> TransactionId xid, Relation rel,
>   pq_sendbyte(out, 'O'); /* old tuple follows */
>   else
>   pq_sendbyte(out, 'K'); /* old key follows */
> - logicalrep_write_tuple(out, rel, oldslot, binary);
> + logicalrep_write_tuple(out, rel, oldslot, binary, columns);
>   }
> 
> As mentioned previously, here, we should pass NULL similar to
> logicalrep_write_delete as we don't need to use column list for old
> tuples.
> 

Fixed.

> 3.
> + * XXX The name is a bit misleading, because we don't really transform
> + * anything here - we merely check the column list is compatible with the
> + * definition of the publication (with publish_via_partition_root=false)
> + * we only allow column lists on the leaf relations. So maybe rename it?
> + */
> +static void
> +TransformPubColumnList(List *tables, const char *queryString,
> +    bool pubviaroot)
> 
> The second parameter is not used in this function. As noted in the
> comments, I also think it is better to rename this. How about
> ValidatePubColumnList?
> 
> 4.
> @@ -821,6 +942,9 @@ fetch_remote_table_info(char *nspname, char *relname,
>   *
>   * 3) one of the subscribed publications is declared as ALL TABLES IN
>   * SCHEMA that includes this relation
> + *
> + * XXX Does this actually handle puballtables and schema publications
> + * correctly?
>   */
>   if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
> 
> Why is this comment added in the row filter code? Now, both row filter
> and column list are fetched in the same way, so not sure what exactly
> this comment is referring to.
> 

I added that comment as a note to myself while learning about how the
code works, forgot to remove that.

> 5.
> +/* qsort comparator for attnums */
> +static int
> +compare_int16(const void *a, const void *b)
> +{
> + int av = *(const int16 *) a;
> + int bv = *(const int16 *) b;
> +
> + /* this can't overflow if int is wider than int16 */
> + return (av - bv);
> +}
> 
> The exact same code exists in statscmds.c. Do we need a second copy of the same?
> 

Yeah, I thought about moving it to some common header, but I think it's
not really worth it at this point.

> 6.
>  static void pgoutput_row_filter_init(PGOutputData *data,
>   List *publications,
>   RelationSyncEntry *entry);
> +
>  static bool pgoutput_row_filter_exec_expr(ExprState *state,
> 
> Spurious line addition.
> 

Fixed.

> 7. The tests in 030_column_list.pl take a long time as compared to all
> other similar individual tests in the subscription folder. I haven't
> checked whether there is any need to reduce some tests but it seems
> worth checking.
> 

On my machine, 'make check' in src/test/subscription takes ~150 seconds
(with asserts and -O0), and the new script takes ~14 seconds, while most
other tests have 3-6 seconds.

AFAICS that's simply due to the number of tests in the script, and I
don't think there are any unnecessary ones. I was actually adding them
in response to issues reported during development, or to test various
important cases. So I don't think we can remove some of them easily :-(

And it's not like the tests are using massive amounts of data either.

We could split the test, but that obviously won't reduce the duration,
of course.

So I decided to keep the test as is, for now, and maybe we can try
reducing the test after a couple buildfarm runs.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  208. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-26T00:35:23Z

    On 3/26/22 01:18, Tomas Vondra wrote:
>
> ...
> 
> I went over the patch again, polished the commit message a bit, and
> pushed. May the buildfarm be merciful!
> 

There's a couple failures immediately after the push, which caused me a
minor heart attack. But it seems all of those are strange failures
related to configure (which the patch did not touch at all), on animals
managed by Andres. And a couple animals succeeded since then.

So I guess the animals were reconfigured, or something ...


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  209. RE: Column Filtering in Logical Replication

    Shinoda, Noriyoshi <noriyoshi.shinoda@hpe.com> — 2022-03-26T04:09:17Z

    Hello, 

The 'prattrs' column has been added to the pg_publication_rel catalog, 
but the current commit to catalog.sgml seems to have added it to pg_publication_namespace. 
The attached patch fixes this.

Regards,
Noriyoshi Shinoda
-----Original Message-----
From: Tomas Vondra <tomas.vondra@enterprisedb.com> 
Sent: Saturday, March 26, 2022 9:35 AM
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>; houzj.fnst@fujitsu.com; Alvaro Herrera <alvherre@alvh.no-ip.org>; Justin Pryzby <pryzby@telsasoft.com>; Rahila Syed <rahilasyed90@gmail.com>; Peter Smith <smithpb2250@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; shiy.fnst@fujitsu.com
Subject: Re: Column Filtering in Logical Replication

On 3/26/22 01:18, Tomas Vondra wrote:
>
> ...
> 
> I went over the patch again, polished the commit message a bit, and 
> pushed. May the buildfarm be merciful!
> 

There's a couple failures immediately after the push, which caused me a minor heart attack. But it seems all of those are strange failures related to configure (which the patch did not touch at all), on animals managed by Andres. And a couple animals succeeded since then.

So I guess the animals were reconfigured, or something ...


regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  210. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-26T09:58:05Z

    On 3/26/22 05:09, Shinoda, Noriyoshi (PN Japan FSIP) wrote:
> Hello, 
> 
> The 'prattrs' column has been added to the pg_publication_rel catalog, 
> but the current commit to catalog.sgml seems to have added it to pg_publication_namespace. 
> The attached patch fixes this.
> 

Thanks, I'll get this pushed.

Sadly, while looking at the catalog docs I realized I forgot to bump the
catversion :-(


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  211. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-26T18:15:37Z

    On 3/26/22 10:58, Tomas Vondra wrote:
> On 3/26/22 05:09, Shinoda, Noriyoshi (PN Japan FSIP) wrote:
>> Hello, 
>>
>> The 'prattrs' column has been added to the pg_publication_rel catalog, 
>> but the current commit to catalog.sgml seems to have added it to pg_publication_namespace. 
>> The attached patch fixes this.
>>
> 
> Thanks, I'll get this pushed.
> 

Pushed. Thanks for noticing this!


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  212. Re: Column Filtering in Logical Replication

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-03-26T21:37:14Z

    Tomas Vondra <tomas.vondra@enterprisedb.com> writes:
> I went over the patch again, polished the commit message a bit, and
> pushed. May the buildfarm be merciful!

Initial results aren't that great.  komodoensis[1], petalura[2],
and snapper[3] have all shown variants of

#   Failed test 'partitions with different replica identities not replicated correctly'
#   at t/031_column_list.pl line 734.
#          got: '2|4|
# 4|9|'
#     expected: '1||5
# 2|4|
# 3||8
# 4|9|'
# Looks like you failed 1 test of 34.
[18:19:36] t/031_column_list.pl ............... 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/34 subtests 

snapper reported different actual output than the other two:
#          got: '1||5
# 3||8'

The failure seems intermittent, as both komodoensis and petalura
have also passed cleanly since the commit (snapper's only run once).

This smells like an uninitialized-variable problem, but I've had
no luck finding any problem under valgrind.  Not sure how to progress
from here.

			regards, tom lane

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=komodoensis&dt=2022-03-26%2015%3A54%3A04
[2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=petalura&dt=2022-03-26%2004%3A20%3A04
[3] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=snapper&dt=2022-03-26%2018%3A46%3A28

  213. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-26T21:52:53Z

    On 3/26/22 22:37, Tom Lane wrote:
> Tomas Vondra <tomas.vondra@enterprisedb.com> writes:
>> I went over the patch again, polished the commit message a bit, and
>> pushed. May the buildfarm be merciful!
> 
> Initial results aren't that great.  komodoensis[1], petalura[2],
> and snapper[3] have all shown variants of
> 
> #   Failed test 'partitions with different replica identities not replicated correctly'
> #   at t/031_column_list.pl line 734.
> #          got: '2|4|
> # 4|9|'
> #     expected: '1||5
> # 2|4|
> # 3||8
> # 4|9|'
> # Looks like you failed 1 test of 34.
> [18:19:36] t/031_column_list.pl ............... 
> Dubious, test returned 1 (wstat 256, 0x100)
> Failed 1/34 subtests 
> 
> snapper reported different actual output than the other two:
> #          got: '1||5
> # 3||8'
> 
> The failure seems intermittent, as both komodoensis and petalura
> have also passed cleanly since the commit (snapper's only run once).
> 
> This smells like an uninitialized-variable problem, but I've had
> no luck finding any problem under valgrind.  Not sure how to progress
> from here.
> 

I think I see the problem - there's a CREATE SUBSCRIPTION but the test
is not waiting for the tablesync to complete, so sometimes it finishes
in time and sometimes not. That'd explain the flaky behavior, and it's
just this one test that misses the sync AFAICS.

FWIW I did run this under valgrind a number of times, and also on
various ARM machines that tend to trip over memory issues.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  214. Re: Column Filtering in Logical Replication

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-03-26T21:55:55Z

    Tomas Vondra <tomas.vondra@enterprisedb.com> writes:
> On 3/26/22 22:37, Tom Lane wrote:
>> This smells like an uninitialized-variable problem, but I've had
>> no luck finding any problem under valgrind.  Not sure how to progress
>> from here.

> I think I see the problem - there's a CREATE SUBSCRIPTION but the test
> is not waiting for the tablesync to complete, so sometimes it finishes
> in time and sometimes not. That'd explain the flaky behavior, and it's
> just this one test that misses the sync AFAICS.

Ah, that would also fit the symptoms.

			regards, tom lane

  215. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-26T21:58:01Z

    On 3/26/22 22:55, Tom Lane wrote:
> Tomas Vondra <tomas.vondra@enterprisedb.com> writes:
>> On 3/26/22 22:37, Tom Lane wrote:
>>> This smells like an uninitialized-variable problem, but I've had
>>> no luck finding any problem under valgrind.  Not sure how to progress
>>> from here.
> 
>> I think I see the problem - there's a CREATE SUBSCRIPTION but the test
>> is not waiting for the tablesync to complete, so sometimes it finishes
>> in time and sometimes not. That'd explain the flaky behavior, and it's
>> just this one test that misses the sync AFAICS.
> 
> Ah, that would also fit the symptoms.
> 

I'll go over the test to check if some other test misses that, and
perhaps do a bit of testing, and then push a fix.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  216. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-26T23:39:52Z

    On 3/26/22 22:58, Tomas Vondra wrote:
> On 3/26/22 22:55, Tom Lane wrote:
>> Tomas Vondra <tomas.vondra@enterprisedb.com> writes:
>>> On 3/26/22 22:37, Tom Lane wrote:
>>>> This smells like an uninitialized-variable problem, but I've had
>>>> no luck finding any problem under valgrind.  Not sure how to progress
>>>> from here.
>>
>>> I think I see the problem - there's a CREATE SUBSCRIPTION but the test
>>> is not waiting for the tablesync to complete, so sometimes it finishes
>>> in time and sometimes not. That'd explain the flaky behavior, and it's
>>> just this one test that misses the sync AFAICS.
>>
>> Ah, that would also fit the symptoms.
>>
> 
> I'll go over the test to check if some other test misses that, and
> perhaps do a bit of testing, and then push a fix.
> 

Pushed. I checked the other tests in 031_column_list.pl and I AFAICS all
of them are waiting for the sync correctly.


[rolls eyes] I just noticed I listed the file as .sql in the commit
message. Not great.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  217. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-29T10:00:22Z

    On Sun, Mar 20, 2022 at 4:53 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/20/22 07:23, Amit Kapila wrote:
> > On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >>
> >> On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
> >> <tomas.vondra@enterprisedb.com> wrote:
> >>
> >>> So the question is why those two sync workers never complete - I guess
> >>> there's some sort of lock wait (deadlock?) or infinite loop.
> >>>
> >>
> >> It would be a bit tricky to reproduce this even if the above theory is
> >> correct but I'll try it today or tomorrow.
> >>
> >
> > I am able to reproduce it with the help of a debugger. Firstly, I have
> > added the LOG message and some While (true) loops to debug sync and
> > apply workers. Test setup
> >
> > Node-1:
> > create table t1(c1);
> > create table t2(c1);
> > insert into t1 values(1);
> > create publication pub1 for table t1;
> > create publication pu2;
> >
> > Node-2:
> > change max_sync_workers_per_subscription to 1 in potgresql.conf
> > create table t1(c1);
> > create table t2(c1);
> > create subscription sub1 connection 'dbname = postgres' publication pub1;
> >
> > Till this point, just allow debuggers in both workers just continue.
> >
> > Node-1:
> > alter publication pub1 add table t2;
> > insert into t1 values(2);
> >
> > Here, we have to debug the apply worker such that when it tries to
> > apply the insert, stop the debugger in function apply_handle_insert()
> > after doing begin_replication_step().
> >
> > Node-2:
> > alter subscription sub1 set pub1, pub2;
> >
> > Now, continue the debugger of apply worker, it should first start the
> > sync worker and then exit because of parameter change. All of these
> > debugging steps are to just ensure the point that it should first
> > start the sync worker and then exit. After this point, table sync
> > worker never finishes and log is filled with messages: "reached
> > max_sync_workers_per_subscription limit" (a newly added message by me
> > in the attached debug patch).
> >
> > Now, it is not completely clear to me how exactly '013_partition.pl'
> > leads to this situation but there is a possibility based on the LOGs
> > it shows.
> >
>
> Thanks, I'll take a look later.
>

This is still failing [1][2].

[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08

-- 
With Regards,
Amit Kapila.

  218. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-29T11:03:34Z

    
On 3/29/22 12:00, Amit Kapila wrote:
> On Sun, Mar 20, 2022 at 4:53 PM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> On 3/20/22 07:23, Amit Kapila wrote:
>>> On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>>>
>>>> On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
>>>> <tomas.vondra@enterprisedb.com> wrote:
>>>>
>>>>> So the question is why those two sync workers never complete - I guess
>>>>> there's some sort of lock wait (deadlock?) or infinite loop.
>>>>>
>>>>
>>>> It would be a bit tricky to reproduce this even if the above theory is
>>>> correct but I'll try it today or tomorrow.
>>>>
>>>
>>> I am able to reproduce it with the help of a debugger. Firstly, I have
>>> added the LOG message and some While (true) loops to debug sync and
>>> apply workers. Test setup
>>>
>>> Node-1:
>>> create table t1(c1);
>>> create table t2(c1);
>>> insert into t1 values(1);
>>> create publication pub1 for table t1;
>>> create publication pu2;
>>>
>>> Node-2:
>>> change max_sync_workers_per_subscription to 1 in potgresql.conf
>>> create table t1(c1);
>>> create table t2(c1);
>>> create subscription sub1 connection 'dbname = postgres' publication pub1;
>>>
>>> Till this point, just allow debuggers in both workers just continue.
>>>
>>> Node-1:
>>> alter publication pub1 add table t2;
>>> insert into t1 values(2);
>>>
>>> Here, we have to debug the apply worker such that when it tries to
>>> apply the insert, stop the debugger in function apply_handle_insert()
>>> after doing begin_replication_step().
>>>
>>> Node-2:
>>> alter subscription sub1 set pub1, pub2;
>>>
>>> Now, continue the debugger of apply worker, it should first start the
>>> sync worker and then exit because of parameter change. All of these
>>> debugging steps are to just ensure the point that it should first
>>> start the sync worker and then exit. After this point, table sync
>>> worker never finishes and log is filled with messages: "reached
>>> max_sync_workers_per_subscription limit" (a newly added message by me
>>> in the attached debug patch).
>>>
>>> Now, it is not completely clear to me how exactly '013_partition.pl'
>>> leads to this situation but there is a possibility based on the LOGs
>>> it shows.
>>>
>>
>> Thanks, I'll take a look later.
>>
> 
> This is still failing [1][2].
> 
> [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
> [2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08
> 

AFAICS we've concluded this is a pre-existing issue, not something
introduced by a recently committed patch, and I don't think there's any
proposal how to fix that. So I've put that on the back burner until
after the current CF.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  219. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-29T11:47:18Z

    On Tue, Mar 29, 2022 at 4:33 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/29/22 12:00, Amit Kapila wrote:
> >>
> >> Thanks, I'll take a look later.
> >>
> >
> > This is still failing [1][2].
> >
> > [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
> > [2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08
> >
>
> AFAICS we've concluded this is a pre-existing issue, not something
> introduced by a recently committed patch, and I don't think there's any
> proposal how to fix that.
>

I have suggested in email [1] that increasing values
max_sync_workers_per_subscription/max_logical_replication_workers
should solve this issue. Now, whether this is a previous issue or
behavior can be debatable but I think it happens for the new test case
added by commit c91f71b9dc.

> So I've put that on the back burner until
> after the current CF.
>

Okay, last time you didn't mention that you want to look at it after
CF. I just assumed that you want to take a look after pushing the main
column list patch, so thought of sending a reminder but I am fine if
you want to look at it after CF.

[1] - https://www.postgresql.org/message-id/CAA4eK1LpBFU49Ohbnk%3Ddv_v9YP%2BKqh1%2BSf8i%2B%2B_s-QhD1Gy4Qw%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  220. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-29T12:39:37Z

    
On 3/29/22 13:47, Amit Kapila wrote:
> On Tue, Mar 29, 2022 at 4:33 PM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> On 3/29/22 12:00, Amit Kapila wrote:
>>>>
>>>> Thanks, I'll take a look later.
>>>>
>>>
>>> This is still failing [1][2].
>>>
>>> [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
>>> [2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08
>>>
>>
>> AFAICS we've concluded this is a pre-existing issue, not something
>> introduced by a recently committed patch, and I don't think there's any
>> proposal how to fix that.
>>
> 
> I have suggested in email [1] that increasing values
> max_sync_workers_per_subscription/max_logical_replication_workers
> should solve this issue. Now, whether this is a previous issue or
> behavior can be debatable but I think it happens for the new test case
> added by commit c91f71b9dc.
> 

IMHO that'd be just hiding the actual issue, which is the failure to
sync the subscription in some circumstances. We should fix that, not
just make sure the tests don't trigger it.

>> So I've put that on the back burner until
>> after the current CF.
>>
> 
> Okay, last time you didn't mention that you want to look at it after
> CF. I just assumed that you want to take a look after pushing the main
> column list patch, so thought of sending a reminder but I am fine if
> you want to look at it after CF.
> 

OK, sorry for not being clearer in my response.

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  221. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-30T02:46:03Z

    On Tue, Mar 29, 2022 at 6:09 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 3/29/22 13:47, Amit Kapila wrote:
> > On Tue, Mar 29, 2022 at 4:33 PM Tomas Vondra
> > <tomas.vondra@enterprisedb.com> wrote:
> >>
> >> On 3/29/22 12:00, Amit Kapila wrote:
> >>>>
> >>>> Thanks, I'll take a look later.
> >>>>
> >>>
> >>> This is still failing [1][2].
> >>>
> >>> [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
> >>> [2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08
> >>>
> >>
> >> AFAICS we've concluded this is a pre-existing issue, not something
> >> introduced by a recently committed patch, and I don't think there's any
> >> proposal how to fix that.
> >>
> >
> > I have suggested in email [1] that increasing values
> > max_sync_workers_per_subscription/max_logical_replication_workers
> > should solve this issue. Now, whether this is a previous issue or
> > behavior can be debatable but I think it happens for the new test case
> > added by commit c91f71b9dc.
> >
>
> IMHO that'd be just hiding the actual issue, which is the failure to
> sync the subscription in some circumstances. We should fix that, not
> just make sure the tests don't trigger it.
>

I am in favor of fixing/changing some existing behavior to make it
better and would be ready to help in that investigation as well but
was just not sure if it is a good idea to let some of the buildfarm
member(s) fail for a number of days. Anyway, I leave this judgment to
you.

-- 
With Regards,
Amit Kapila.

  222. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-03-30T12:01:38Z

    
On 3/30/22 04:46, Amit Kapila wrote:
> On Tue, Mar 29, 2022 at 6:09 PM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> On 3/29/22 13:47, Amit Kapila wrote:
>>> On Tue, Mar 29, 2022 at 4:33 PM Tomas Vondra
>>> <tomas.vondra@enterprisedb.com> wrote:
>>>>
>>>> On 3/29/22 12:00, Amit Kapila wrote:
>>>>>>
>>>>>> Thanks, I'll take a look later.
>>>>>>
>>>>>
>>>>> This is still failing [1][2].
>>>>>
>>>>> [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
>>>>> [2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08
>>>>>
>>>>
>>>> AFAICS we've concluded this is a pre-existing issue, not something
>>>> introduced by a recently committed patch, and I don't think there's any
>>>> proposal how to fix that.
>>>>
>>>
>>> I have suggested in email [1] that increasing values
>>> max_sync_workers_per_subscription/max_logical_replication_workers
>>> should solve this issue. Now, whether this is a previous issue or
>>> behavior can be debatable but I think it happens for the new test case
>>> added by commit c91f71b9dc.
>>>
>>
>> IMHO that'd be just hiding the actual issue, which is the failure to
>> sync the subscription in some circumstances. We should fix that, not
>> just make sure the tests don't trigger it.
>>
> 
> I am in favor of fixing/changing some existing behavior to make it
> better and would be ready to help in that investigation as well but
> was just not sure if it is a good idea to let some of the buildfarm
> member(s) fail for a number of days. Anyway, I leave this judgment to
> you.
> 

OK. If it affected more animals, and/or if they were failing more often,
it'd definitely warrant a more active approach. But AFAICS it affects
only a tiny fraction, and even there it fails maybe 1 in 20 runs ...

Plus the symptoms are pretty clear, it's unlikely to cause enigmatic
failures, forcing people to spend time on investigating it.

Of course, that's my assessment and it feels weird as it goes directly
against my instincts to keep all tests working :-/


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  223. Re: Column Filtering in Logical Replication

    Masahiko Sawada <sawada.mshk@gmail.com> — 2022-04-13T08:10:27Z

    On Sun, Mar 20, 2022 at 3:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
> > <tomas.vondra@enterprisedb.com> wrote:
> >
> > > So the question is why those two sync workers never complete - I guess
> > > there's some sort of lock wait (deadlock?) or infinite loop.
> > >
> >
> > It would be a bit tricky to reproduce this even if the above theory is
> > correct but I'll try it today or tomorrow.
> >
>
> I am able to reproduce it with the help of a debugger. Firstly, I have
> added the LOG message and some While (true) loops to debug sync and
> apply workers. Test setup
>
> Node-1:
> create table t1(c1);
> create table t2(c1);
> insert into t1 values(1);
> create publication pub1 for table t1;
> create publication pu2;
>
> Node-2:
> change max_sync_workers_per_subscription to 1 in potgresql.conf
> create table t1(c1);
> create table t2(c1);
> create subscription sub1 connection 'dbname = postgres' publication pub1;
>
> Till this point, just allow debuggers in both workers just continue.
>
> Node-1:
> alter publication pub1 add table t2;
> insert into t1 values(2);
>
> Here, we have to debug the apply worker such that when it tries to
> apply the insert, stop the debugger in function apply_handle_insert()
> after doing begin_replication_step().
>
> Node-2:
> alter subscription sub1 set pub1, pub2;
>
> Now, continue the debugger of apply worker, it should first start the
> sync worker and then exit because of parameter change. All of these
> debugging steps are to just ensure the point that it should first
> start the sync worker and then exit. After this point, table sync
> worker never finishes and log is filled with messages: "reached
> max_sync_workers_per_subscription limit" (a newly added message by me
> in the attached debug patch).
>
> Now, it is not completely clear to me how exactly '013_partition.pl'
> leads to this situation but there is a possibility based on the LOGs

I've looked at this issue and had the same analysis. Also, I could
reproduce this issue with the steps shared by Amit.

As I mentioned in another thread[1], the fact that the tablesync
worker doesn't check the return value from
wait_for_worker_state_change() seems a bug to me. So my initial
thought of the solution is that we can have the tablesync worker check
the return value and exit if it's false. That way, the apply worker
can restart and request to launch the tablesync worker again. What do
you think?

Regards,

-- 
Masahiko Sawada
EDB:  https://www.enterprisedb.com/

  224. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-04-13T09:45:41Z

    On Wed, Apr 13, 2022 at 1:41 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> I've looked at this issue and had the same analysis. Also, I could
> reproduce this issue with the steps shared by Amit.
>
> As I mentioned in another thread[1], the fact that the tablesync
> worker doesn't check the return value from
> wait_for_worker_state_change() seems a bug to me. So my initial
> thought of the solution is that we can have the tablesync worker check
> the return value and exit if it's false. That way, the apply worker
> can restart and request to launch the tablesync worker again. What do
> you think?
>

I think that will fix this symptom but I am not sure if that would be
the best way to deal with this because we have a mechanism where the
sync worker can continue even if we don't do anything as a result of
wait_for_worker_state_change() provided apply worker restarts.

The other part of the puzzle is the below check in the code:
/*
* If we reached the sync worker limit per subscription, just exit
* silently as we might get here because of an otherwise harmless race
* condition.
*/
if (nsyncworkers >= max_sync_workers_per_subscription)

It is not clear to me why this check is there, if this wouldn't be
there, the user would have got either a WARNING to increase the
max_logical_replication_workers or the apply worker would have been
restarted. Do you have any idea about this?

Yet another option is that we ensure that before launching sync
workers (say in process_syncing_tables_for_apply->FetchTableStates,
when we have to start a new transaction) we again call
maybe_reread_subscription(), which should also fix this symptom. But
again, I am not sure why it should be compulsory to call
maybe_reread_subscription() in such a situation, there are no comments
which suggest it,

Now, the reason why it appeared recently in commit c91f71b9dc is that
I think we have increased the number of initial table syncs in that
test, and probably increasing
max_sync_workers_per_subscription/max_logical_replication_workers
should fix that test.

-- 
With Regards,
Amit Kapila.

  225. Re: Column Filtering in Logical Replication

    Masahiko Sawada <sawada.mshk@gmail.com> — 2022-04-14T03:02:16Z

    (

On Wed, Apr 13, 2022 at 6:45 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Apr 13, 2022 at 1:41 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > I've looked at this issue and had the same analysis. Also, I could
> > reproduce this issue with the steps shared by Amit.
> >
> > As I mentioned in another thread[1], the fact that the tablesync
> > worker doesn't check the return value from
> > wait_for_worker_state_change() seems a bug to me. So my initial
> > thought of the solution is that we can have the tablesync worker check
> > the return value and exit if it's false. That way, the apply worker
> > can restart and request to launch the tablesync worker again. What do
> > you think?
> >
>
> I think that will fix this symptom but I am not sure if that would be
> the best way to deal with this because we have a mechanism where the
> sync worker can continue even if we don't do anything as a result of
> wait_for_worker_state_change() provided apply worker restarts.

I think we can think this is a separate issue. That is, if tablesync
worker can start streaming changes even without waiting for the apply
worker to set SUBREL_STATE_CATCHUP, do we really need the wait? I'm
not sure it's really safe. If it's safe, the tablesync worker will no
longer need to wait there.

>
> The other part of the puzzle is the below check in the code:
> /*
> * If we reached the sync worker limit per subscription, just exit
> * silently as we might get here because of an otherwise harmless race
> * condition.
> */
> if (nsyncworkers >= max_sync_workers_per_subscription)
>
> It is not clear to me why this check is there, if this wouldn't be
> there, the user would have got either a WARNING to increase the
> max_logical_replication_workers or the apply worker would have been
> restarted. Do you have any idea about this?

Yeah, I'm also puzzled with this check. It seems that this function
doesn't work well when the apply worker is not running and some
tablesync workers are running. I initially thought that the apply
worker calls to this function as many as tables that needs to be
synced, but it checks the max_sync_workers_per_subscription limit
before calling to logicalrep_worker_launch(). So I'm not really sure
we need this check.

>
> Yet another option is that we ensure that before launching sync
> workers (say in process_syncing_tables_for_apply->FetchTableStates,
> when we have to start a new transaction) we again call
> maybe_reread_subscription(), which should also fix this symptom. But
> again, I am not sure why it should be compulsory to call
> maybe_reread_subscription() in such a situation, there are no comments
> which suggest it,

Yes, it will fix this issue.

>
> Now, the reason why it appeared recently in commit c91f71b9dc is that
> I think we have increased the number of initial table syncs in that
> test, and probably increasing
> max_sync_workers_per_subscription/max_logical_replication_workers
> should fix that test.

I think so too.

Regards,

-- 
Masahiko Sawada
EDB:  https://www.enterprisedb.com/

  226. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-04-14T03:39:31Z

    On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Apr 13, 2022 at 6:45 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Apr 13, 2022 at 1:41 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > I've looked at this issue and had the same analysis. Also, I could
> > > reproduce this issue with the steps shared by Amit.
> > >
> > > As I mentioned in another thread[1], the fact that the tablesync
> > > worker doesn't check the return value from
> > > wait_for_worker_state_change() seems a bug to me. So my initial
> > > thought of the solution is that we can have the tablesync worker check
> > > the return value and exit if it's false. That way, the apply worker
> > > can restart and request to launch the tablesync worker again. What do
> > > you think?
> > >
> >
> > I think that will fix this symptom but I am not sure if that would be
> > the best way to deal with this because we have a mechanism where the
> > sync worker can continue even if we don't do anything as a result of
> > wait_for_worker_state_change() provided apply worker restarts.
>
> I think we can think this is a separate issue. That is, if tablesync
> worker can start streaming changes even without waiting for the apply
> worker to set SUBREL_STATE_CATCHUP, do we really need the wait? I'm
> not sure it's really safe. If it's safe, the tablesync worker will no
> longer need to wait there.
>

As per my understanding, it is safe, whatever is streamed by tablesync
worker will be skipped later by apply worker. The wait here avoids
streaming the same data both by the apply worker and table sync worker
which I think is good even if it is not a must.

> >
> > The other part of the puzzle is the below check in the code:
> > /*
> > * If we reached the sync worker limit per subscription, just exit
> > * silently as we might get here because of an otherwise harmless race
> > * condition.
> > */
> > if (nsyncworkers >= max_sync_workers_per_subscription)
> >
> > It is not clear to me why this check is there, if this wouldn't be
> > there, the user would have got either a WARNING to increase the
> > max_logical_replication_workers or the apply worker would have been
> > restarted. Do you have any idea about this?
>
> Yeah, I'm also puzzled with this check. It seems that this function
> doesn't work well when the apply worker is not running and some
> tablesync workers are running. I initially thought that the apply
> worker calls to this function as many as tables that needs to be
> synced, but it checks the max_sync_workers_per_subscription limit
> before calling to logicalrep_worker_launch(). So I'm not really sure
> we need this check.
>

I just hope that the original author Petr J. responds to this point. I
have added him to this email. This will help us to find the best
solution for this problem.

Note: I'll be away for the remaining week, so will join the discussion
next week unless we reached the conclusion by that time.

-- 
With Regards,
Amit Kapila.

  227. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-04-18T11:04:06Z

    On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > The other part of the puzzle is the below check in the code:
> > > /*
> > > * If we reached the sync worker limit per subscription, just exit
> > > * silently as we might get here because of an otherwise harmless race
> > > * condition.
> > > */
> > > if (nsyncworkers >= max_sync_workers_per_subscription)
> > >
> > > It is not clear to me why this check is there, if this wouldn't be
> > > there, the user would have got either a WARNING to increase the
> > > max_logical_replication_workers or the apply worker would have been
> > > restarted. Do you have any idea about this?
> >
> > Yeah, I'm also puzzled with this check. It seems that this function
> > doesn't work well when the apply worker is not running and some
> > tablesync workers are running. I initially thought that the apply
> > worker calls to this function as many as tables that needs to be
> > synced, but it checks the max_sync_workers_per_subscription limit
> > before calling to logicalrep_worker_launch(). So I'm not really sure
> > we need this check.
> >
>
> I just hope that the original author Petr J. responds to this point. I
> have added him to this email. This will help us to find the best
> solution for this problem.
>

I did some more investigation for this code. It is added by commit [1]
and the patch that led to this commit is first time posted on -hackers
in email [2]. Now, neither the commit message nor the patch (comments)
gives much idea as to why this part of code is added but I think there
is some hint in the email [2]. In particular, read the paragraph in
the email [2] that has the lines: ".... and limiting sync workers per
subscription theoretically wasn't either (although I don't think it
could happen in practice).".

It seems that this check has been added to theoretically limit the
sync workers even though that can't happen because apply worker
ensures that before trying to launch the sync worker. Does this theory
make sense to me? If so, I think we can change the check as: "if
(OidIsValid(relid) && nsyncworkers >=
max_sync_workers_per_subscription)" in launcher.c. This will serve the
purpose of the original code and will solve the issue being discussed
here. I think we can even backpatch this. What do you think?

[1]
commit de4389712206d2686e09ad8d6dd112dc4b6c6d42
Author: Peter Eisentraut <peter_e@gmx.net>
Date:   Wed Apr 26 10:43:04 2017 -0400

    Fix various concurrency issues in logical replication worker launching

[2] - https://www.postgresql.org/message-id/fa387e24-0e26-c02d-ef16-7e46ada200dd%402ndquadrant.com

-- 
With Regards,
Amit Kapila.

  228. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-04-18T12:36:00Z

    On 4/18/22 13:04, Amit Kapila wrote:
> On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>>>
>>>> The other part of the puzzle is the below check in the code:
>>>> /*
>>>> * If we reached the sync worker limit per subscription, just exit
>>>> * silently as we might get here because of an otherwise harmless race
>>>> * condition.
>>>> */
>>>> if (nsyncworkers >= max_sync_workers_per_subscription)
>>>>
>>>> It is not clear to me why this check is there, if this wouldn't be
>>>> there, the user would have got either a WARNING to increase the
>>>> max_logical_replication_workers or the apply worker would have been
>>>> restarted. Do you have any idea about this?
>>>
>>> Yeah, I'm also puzzled with this check. It seems that this function
>>> doesn't work well when the apply worker is not running and some
>>> tablesync workers are running. I initially thought that the apply
>>> worker calls to this function as many as tables that needs to be
>>> synced, but it checks the max_sync_workers_per_subscription limit
>>> before calling to logicalrep_worker_launch(). So I'm not really sure
>>> we need this check.
>>>
>>
>> I just hope that the original author Petr J. responds to this point. I
>> have added him to this email. This will help us to find the best
>> solution for this problem.
>>
> 
> I did some more investigation for this code. It is added by commit [1]
> and the patch that led to this commit is first time posted on -hackers
> in email [2]. Now, neither the commit message nor the patch (comments)
> gives much idea as to why this part of code is added but I think there
> is some hint in the email [2]. In particular, read the paragraph in
> the email [2] that has the lines: ".... and limiting sync workers per
> subscription theoretically wasn't either (although I don't think it
> could happen in practice).".
> 
> It seems that this check has been added to theoretically limit the
> sync workers even though that can't happen because apply worker
> ensures that before trying to launch the sync worker. Does this theory
> make sense to me? If so, I think we can change the check as: "if
> (OidIsValid(relid) && nsyncworkers >=
> max_sync_workers_per_subscription)" in launcher.c. This will serve the
> purpose of the original code and will solve the issue being discussed
> here. I think we can even backpatch this. What do you think?
> 

Sounds reasonable to me. It's unfortunate there's no explanation of what
exactly is the commit message fixing (and why), but I doubt anyone will
remember the details after 5 years.

+1 to backpatching, I consider this to be a bug


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  229. Re: Column Filtering in Logical Replication

    Masahiko Sawada <sawada.mshk@gmail.com> — 2022-04-19T01:27:40Z

    On Mon, Apr 18, 2022 at 8:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > The other part of the puzzle is the below check in the code:
> > > > /*
> > > > * If we reached the sync worker limit per subscription, just exit
> > > > * silently as we might get here because of an otherwise harmless race
> > > > * condition.
> > > > */
> > > > if (nsyncworkers >= max_sync_workers_per_subscription)
> > > >
> > > > It is not clear to me why this check is there, if this wouldn't be
> > > > there, the user would have got either a WARNING to increase the
> > > > max_logical_replication_workers or the apply worker would have been
> > > > restarted. Do you have any idea about this?
> > >
> > > Yeah, I'm also puzzled with this check. It seems that this function
> > > doesn't work well when the apply worker is not running and some
> > > tablesync workers are running. I initially thought that the apply
> > > worker calls to this function as many as tables that needs to be
> > > synced, but it checks the max_sync_workers_per_subscription limit
> > > before calling to logicalrep_worker_launch(). So I'm not really sure
> > > we need this check.
> > >
> >
> > I just hope that the original author Petr J. responds to this point. I
> > have added him to this email. This will help us to find the best
> > solution for this problem.
> >
>
> I did some more investigation for this code. It is added by commit [1]
> and the patch that led to this commit is first time posted on -hackers
> in email [2]. Now, neither the commit message nor the patch (comments)
> gives much idea as to why this part of code is added but I think there
> is some hint in the email [2]. In particular, read the paragraph in
> the email [2] that has the lines: ".... and limiting sync workers per
> subscription theoretically wasn't either (although I don't think it
> could happen in practice).".
>
> It seems that this check has been added to theoretically limit the
> sync workers even though that can't happen because apply worker
> ensures that before trying to launch the sync worker. Does this theory
> make sense to me? If so, I think we can change the check as: "if
> (OidIsValid(relid) && nsyncworkers >=
> max_sync_workers_per_subscription)" in launcher.c. This will serve the
> purpose of the original code and will solve the issue being discussed
> here. I think we can even backpatch this. What do you think?

+1. I also think it's a bug so back-patching makes sense to me.

Regards,

-- 
Masahiko Sawada
EDB:  https://www.enterprisedb.com/

  230. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-04-19T04:53:22Z

    On Tue, Apr 19, 2022 at 6:58 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Apr 18, 2022 at 8:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > The other part of the puzzle is the below check in the code:
> > > > > /*
> > > > > * If we reached the sync worker limit per subscription, just exit
> > > > > * silently as we might get here because of an otherwise harmless race
> > > > > * condition.
> > > > > */
> > > > > if (nsyncworkers >= max_sync_workers_per_subscription)
> > > > >
> > > > > It is not clear to me why this check is there, if this wouldn't be
> > > > > there, the user would have got either a WARNING to increase the
> > > > > max_logical_replication_workers or the apply worker would have been
> > > > > restarted. Do you have any idea about this?
> > > >
> > > > Yeah, I'm also puzzled with this check. It seems that this function
> > > > doesn't work well when the apply worker is not running and some
> > > > tablesync workers are running. I initially thought that the apply
> > > > worker calls to this function as many as tables that needs to be
> > > > synced, but it checks the max_sync_workers_per_subscription limit
> > > > before calling to logicalrep_worker_launch(). So I'm not really sure
> > > > we need this check.
> > > >
> > >
> > > I just hope that the original author Petr J. responds to this point. I
> > > have added him to this email. This will help us to find the best
> > > solution for this problem.
> > >
> >
> > I did some more investigation for this code. It is added by commit [1]
> > and the patch that led to this commit is first time posted on -hackers
> > in email [2]. Now, neither the commit message nor the patch (comments)
> > gives much idea as to why this part of code is added but I think there
> > is some hint in the email [2]. In particular, read the paragraph in
> > the email [2] that has the lines: ".... and limiting sync workers per
> > subscription theoretically wasn't either (although I don't think it
> > could happen in practice).".
> >
> > It seems that this check has been added to theoretically limit the
> > sync workers even though that can't happen because apply worker
> > ensures that before trying to launch the sync worker. Does this theory
> > make sense to me? If so, I think we can change the check as: "if
> > (OidIsValid(relid) && nsyncworkers >=
> > max_sync_workers_per_subscription)" in launcher.c. This will serve the
> > purpose of the original code and will solve the issue being discussed
> > here. I think we can even backpatch this. What do you think?
>
> +1. I also think it's a bug so back-patching makes sense to me.
>

Pushed. Thanks Tomas and Sawada-San.

-- 
With Regards,
Amit Kapila.

  231. Re: Column Filtering in Logical Replication

    Jonathan S. Katz <jkatz@postgresql.org> — 2022-05-10T13:55:18Z

    Hi,

On 4/19/22 12:53 AM, Amit Kapila wrote:
> On Tue, Apr 19, 2022 at 6:58 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>
>> +1. I also think it's a bug so back-patching makes sense to me.
>>
> 
> Pushed. Thanks Tomas and Sawada-San.

This is still on the PG15 open items list[1] though marked as with a fix.

Did dd4ab6fd resolve the issue, or does this need more work?

Thanks,

Jonathan

[1] https://wiki.postgresql.org/wiki/PostgreSQL_15_Open_Items

  232. Re: Column Filtering in Logical Replication

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2022-05-10T19:17:45Z

    On 5/10/22 15:55, Jonathan S. Katz wrote:
> Hi,
> 
> On 4/19/22 12:53 AM, Amit Kapila wrote:
>> On Tue, Apr 19, 2022 at 6:58 AM Masahiko Sawada
>> <sawada.mshk@gmail.com> wrote:
>>>
>>> +1. I also think it's a bug so back-patching makes sense to me.
>>>
>>
>> Pushed. Thanks Tomas and Sawada-San.
> 
> This is still on the PG15 open items list[1] though marked as with a fix.
> 
> Did dd4ab6fd resolve the issue, or does this need more work?
> 

I believe that's fixed, the buildfarm does not seem to show any relevant
failures in subscriptionCheck since dd4ab6fd got committed.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  233. Re: Column Filtering in Logical Replication

    Jonathan S. Katz <jkatz@postgresql.org> — 2022-05-10T19:28:51Z

    On 5/10/22 3:17 PM, Tomas Vondra wrote:
> On 5/10/22 15:55, Jonathan S. Katz wrote:
>> Hi,
>>
>> On 4/19/22 12:53 AM, Amit Kapila wrote:
>>> On Tue, Apr 19, 2022 at 6:58 AM Masahiko Sawada
>>> <sawada.mshk@gmail.com> wrote:
>>>>
>>>> +1. I also think it's a bug so back-patching makes sense to me.
>>>>
>>>
>>> Pushed. Thanks Tomas and Sawada-San.
>>
>> This is still on the PG15 open items list[1] though marked as with a fix.
>>
>> Did dd4ab6fd resolve the issue, or does this need more work?
>>
> 
> I believe that's fixed, the buildfarm does not seem to show any relevant
> failures in subscriptionCheck since dd4ab6fd got committed.

Great. I'm moving it off of open items.

Thanks for confirming!

Jonathan

  234. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-05-11T03:20:21Z

    On Tue, May 10, 2022 at 7:25 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> On 4/19/22 12:53 AM, Amit Kapila wrote:
> > On Tue, Apr 19, 2022 at 6:58 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >>
> >> +1. I also think it's a bug so back-patching makes sense to me.
> >>
> >
> > Pushed. Thanks Tomas and Sawada-San.
>
> This is still on the PG15 open items list[1] though marked as with a fix.
>
> Did dd4ab6fd resolve the issue, or does this need more work?
>

The commit dd4ab6fd resolved this issue. I didn't notice it after that commit.

-- 
With Regards,
Amit Kapila.

  235. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-07-25T07:57:09Z

    On Sat, Dec 11, 2021 at 12:24 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Dec-10, Peter Eisentraut wrote:
>
...
>
> > There was no documentation, so I wrote a bit (patch 0001).  It only touches
> > the CREATE PUBLICATION and ALTER PUBLICATION pages at the moment.  There was
> > no mention in the Logical Replication chapter that warranted updating.
> > Perhaps we should revisit that chapter at the end of the release cycle.
>
> Thanks.  I hadn't looked at the docs yet, so I'll definitely take this.
>

Was this documentation ever written?

My assumption was that for PG15 there might be a whole new section
added to Chapter 31 [1] for describing "Column Lists" (i.e. the Column
List equivalent of the "Row Filters" section)

------
[1] https://www.postgresql.org/docs/15/logical-replication.html

Kind Regards,
Peter Smith.
Fujitsu Australia

  236. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-02T08:57:38Z

    On Mon, Jul 25, 2022 at 1:27 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Sat, Dec 11, 2021 at 12:24 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> >
> > On 2021-Dec-10, Peter Eisentraut wrote:
> >
> ...
> >
> > > There was no documentation, so I wrote a bit (patch 0001).  It only touches
> > > the CREATE PUBLICATION and ALTER PUBLICATION pages at the moment.  There was
> > > no mention in the Logical Replication chapter that warranted updating.
> > > Perhaps we should revisit that chapter at the end of the release cycle.
> >
> > Thanks.  I hadn't looked at the docs yet, so I'll definitely take this.
> >
>
> Was this documentation ever written?
>
> My assumption was that for PG15 there might be a whole new section
> added to Chapter 31 [1] for describing "Column Lists" (i.e. the Column
> List equivalent of the "Row Filters" section)
>

+1. I think it makes sense to give more description about this feature
similar to Row Filters. Note that apart from the main feature commit
[1], we have prohibited certain cases in commit [2]. So, one might
want to cover that as well.

[1] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=923def9a533a7d986acfb524139d8b9e5466d0a5
[2] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=fd0b9dcebda7b931a41ce5c8e86d13f2efd0af2e

-- 
With Regards,
Amit Kapila.

  237. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-08-02T22:42:01Z

    On Tue, Aug 2, 2022 at 6:57 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jul 25, 2022 at 1:27 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Sat, Dec 11, 2021 at 12:24 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> > >
> > > On 2021-Dec-10, Peter Eisentraut wrote:
> > >
> > ...
> > >
> > > > There was no documentation, so I wrote a bit (patch 0001).  It only touches
> > > > the CREATE PUBLICATION and ALTER PUBLICATION pages at the moment.  There was
> > > > no mention in the Logical Replication chapter that warranted updating.
> > > > Perhaps we should revisit that chapter at the end of the release cycle.
> > >
> > > Thanks.  I hadn't looked at the docs yet, so I'll definitely take this.
> > >
> >
> > Was this documentation ever written?
> >
> > My assumption was that for PG15 there might be a whole new section
> > added to Chapter 31 [1] for describing "Column Lists" (i.e. the Column
> > List equivalent of the "Row Filters" section)
> >
>
> +1. I think it makes sense to give more description about this feature
> similar to Row Filters. Note that apart from the main feature commit
> [1], we have prohibited certain cases in commit [2]. So, one might
> want to cover that as well.
>
> [1] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=923def9a533a7d986acfb524139d8b9e5466d0a5
> [2] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=fd0b9dcebda7b931a41ce5c8e86d13f2efd0af2e
>

OK. Unless somebody else has already started this work then I can do
this. I will post a draft patch in a few days.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  238. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-08-08T08:37:51Z

    PSA patch version v1* for a new "Column Lists" pgdocs section

This is just a first draft, but I wanted to post it as-is, with the
hope that I can get some feedback while continuing to work on it.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  239. Re: Column Filtering in Logical Replication

    vignesh C <vignesh21@gmail.com> — 2022-08-16T17:04:26Z

    On Mon, Aug 8, 2022 at 2:08 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> PSA patch version v1* for a new "Column Lists" pgdocs section
>
> This is just a first draft, but I wanted to post it as-is, with the
> hope that I can get some feedback while continuing to work on it.

Few comments:
1) Row filters mentions that "It has no effect on TRUNCATE commands.",
the same is not present in case of column filters. We should keep the
changes similarly for consistency.
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -90,8 +90,7 @@ CREATE PUBLICATION <replaceable
class="parameter">name</replaceable>
      <para>
       When a column list is specified, only the named columns are replicated.
       If no column list is specified, all columns of the table are replicated
-      through this publication, including any columns added later.  If a column
-      list is specified, it must include the replica identity columns.
+      through this publication, including any columns added later.

2) The document says that "if the table uses REPLICA IDENTITY FULL,
specifying a column list is not allowed.":
+   publishes only <command>INSERT</command> operations. Furthermore, if the
+   table uses <literal>REPLICA IDENTITY FULL</literal>, specifying a column
+   list is not allowed.
+  </para>

Did you mean specifying a column list during create publication for
REPLICA IDENTITY FULL table like below scenario:
postgres=# create table t2(c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# alter table t2 replica identity full ;
ALTER TABLE
postgres=# create publication pub1 for table t2(c1,c2);
CREATE PUBLICATION

If so, the document says specifying column list is not allowed, but
creating a publication with column list on replica identity full was
successful.

Regards,
Vignesh

  240. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-08-22T08:27:43Z

    Thanks for the view of v1-0001.

On Wed, Aug 17, 2022 at 3:04 AM vignesh C <vignesh21@gmail.com> wrote:
...
> 1) Row filters mentions that "It has no effect on TRUNCATE commands.",
> the same is not present in case of column filters. We should keep the
> changes similarly for consistency.
> --- a/doc/src/sgml/ref/create_publication.sgml
> +++ b/doc/src/sgml/ref/create_publication.sgml
> @@ -90,8 +90,7 @@ CREATE PUBLICATION <replaceable
> class="parameter">name</replaceable>
>       <para>
>        When a column list is specified, only the named columns are replicated.
>        If no column list is specified, all columns of the table are replicated
> -      through this publication, including any columns added later.  If a column
> -      list is specified, it must include the replica identity columns.
> +      through this publication, including any columns added later.

Modified as suggested.

>
> 2) The document says that "if the table uses REPLICA IDENTITY FULL,
> specifying a column list is not allowed.":
> +   publishes only <command>INSERT</command> operations. Furthermore, if the
> +   table uses <literal>REPLICA IDENTITY FULL</literal>, specifying a column
> +   list is not allowed.
> +  </para>
>
> Did you mean specifying a column list during create publication for
> REPLICA IDENTITY FULL table like below scenario:
> postgres=# create table t2(c1 int, c2 int, c3 int);
> CREATE TABLE
> postgres=# alter table t2 replica identity full ;
> ALTER TABLE
> postgres=# create publication pub1 for table t2(c1,c2);
> CREATE PUBLICATION
>
> If so, the document says specifying column list is not allowed, but
> creating a publication with column list on replica identity full was
> successful.

That patch v1-0001 was using the same wording from the github commit
message [1]. I agree it was a bit vague.

In fact the replica identity validation is done at DML execution time
so your example will fail as expected when you attempt to do a UPDATE
operation.

e.g.
test_pub=# update t2 set c2=23 where c1=1;
ERROR:  cannot update table "t2"
DETAIL:  Column list used by the publication does not cover the
replica identity.

I modified the wording for this part of the docs.

~~~

PSA new set of v2* patches.

------
[1] - https://github.com/postgres/postgres/commit/923def9a533a7d986acfb524139d8b9e5466d0a5

Kind Regards,
Peter Smith
Fujitsu Australia

  241. Re: Column Filtering in Logical Replication

    Erik Rijkers <er@xs4all.nl> — 2022-08-22T09:11:46Z

    Op 22-08-2022 om 10:27 schreef Peter Smith:
> 
> PSA new set of v2* patches.

Hi,

In the second file a small typo, I think:

"enclosed by parenthesis"  should be
"enclosed by parentheses"

thanks,
Erik

  242. Re: Column Filtering in Logical Replication

    vignesh C <vignesh21@gmail.com> — 2022-08-22T11:25:19Z

    On Mon, Aug 22, 2022 at 1:58 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Thanks for the view of v1-0001.
>
> On Wed, Aug 17, 2022 at 3:04 AM vignesh C <vignesh21@gmail.com> wrote:
> ...
> > 1) Row filters mentions that "It has no effect on TRUNCATE commands.",
> > the same is not present in case of column filters. We should keep the
> > changes similarly for consistency.
> > --- a/doc/src/sgml/ref/create_publication.sgml
> > +++ b/doc/src/sgml/ref/create_publication.sgml
> > @@ -90,8 +90,7 @@ CREATE PUBLICATION <replaceable
> > class="parameter">name</replaceable>
> >       <para>
> >        When a column list is specified, only the named columns are replicated.
> >        If no column list is specified, all columns of the table are replicated
> > -      through this publication, including any columns added later.  If a column
> > -      list is specified, it must include the replica identity columns.
> > +      through this publication, including any columns added later.
>
> Modified as suggested.
>
> >
> > 2) The document says that "if the table uses REPLICA IDENTITY FULL,
> > specifying a column list is not allowed.":
> > +   publishes only <command>INSERT</command> operations. Furthermore, if the
> > +   table uses <literal>REPLICA IDENTITY FULL</literal>, specifying a column
> > +   list is not allowed.
> > +  </para>
> >
> > Did you mean specifying a column list during create publication for
> > REPLICA IDENTITY FULL table like below scenario:
> > postgres=# create table t2(c1 int, c2 int, c3 int);
> > CREATE TABLE
> > postgres=# alter table t2 replica identity full ;
> > ALTER TABLE
> > postgres=# create publication pub1 for table t2(c1,c2);
> > CREATE PUBLICATION
> >
> > If so, the document says specifying column list is not allowed, but
> > creating a publication with column list on replica identity full was
> > successful.
>
> That patch v1-0001 was using the same wording from the github commit
> message [1]. I agree it was a bit vague.
>
> In fact the replica identity validation is done at DML execution time
> so your example will fail as expected when you attempt to do a UPDATE
> operation.
>
> e.g.
> test_pub=# update t2 set c2=23 where c1=1;
> ERROR:  cannot update table "t2"
> DETAIL:  Column list used by the publication does not cover the
> replica identity.
>
> I modified the wording for this part of the docs.

Few comments:
1) I felt no expressions are allowed in case of column filters. Only
column names can be specified. The second part of the sentence
confuses what is allowed and what is not allowed. Won't it be better
to remove the second sentence and mention that only column names can
be specified.
+   <para>
+    Column list can contain only simple column references. Complex
+    expressions, function calls etc. are not allowed.
+   </para>

2) tablename should be table name.
+   <para>
+    A column list is specified per table following the tablename, and
enclosed by
+    parenthesis. See <xref linkend="sql-createpublication"/> for details.
+   </para>

We have used table name in the same page in other instances like:
a) The row filter is defined per table. Use a WHERE clause after the
table name for each published table that requires data to be filtered
out. The WHERE clause must be enclosed by parentheses.
b) The tables are matched between the publisher and the subscriber
using the fully qualified table name.

3) One small whitespace issue:
git am v2-0001-Column-List-replica-identity-rules.patch
Applying: Column List replica identity rules.
.git/rebase-apply/patch:30: trailing whitespace.
   if the publication publishes only <command>INSERT</command> operations.
warning: 1 line adds whitespace errors.

Regards,
Vignesh

  243. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-08-23T02:22:00Z

    On Mon, Aug 22, 2022 at 9:25 PM vignesh C <vignesh21@gmail.com> wrote:
>
...

> Few comments:
> 1) I felt no expressions are allowed in case of column filters. Only
> column names can be specified. The second part of the sentence
> confuses what is allowed and what is not allowed. Won't it be better
> to remove the second sentence and mention that only column names can
> be specified.
> +   <para>
> +    Column list can contain only simple column references. Complex
> +    expressions, function calls etc. are not allowed.
> +   </para>
>

This wording was lifted verbatim from the commit message [1]. But I
see your point that it just seems to be overcomplicating a simple
rule. Modified as suggested.

> 2) tablename should be table name.
> +   <para>
> +    A column list is specified per table following the tablename, and
> enclosed by
> +    parenthesis. See <xref linkend="sql-createpublication"/> for details.
> +   </para>
>
> We have used table name in the same page in other instances like:
> a) The row filter is defined per table. Use a WHERE clause after the
> table name for each published table that requires data to be filtered
> out. The WHERE clause must be enclosed by parentheses.
> b) The tables are matched between the publisher and the subscriber
> using the fully qualified table name.
>

Fixed as suggested.

> 3) One small whitespace issue:
> git am v2-0001-Column-List-replica-identity-rules.patch
> Applying: Column List replica identity rules.
> .git/rebase-apply/patch:30: trailing whitespace.
>    if the publication publishes only <command>INSERT</command> operations.
> warning: 1 line adds whitespace errors.
>

Fixed.

~~~

PSA the v3* patch set.

------
[1] https://github.com/postgres/postgres/commit/923def9a533a7d986acfb524139d8b9e5466d0a5

Kind Regards,
Peter Smith.
Fujitsu Australia

  244. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-08-23T02:25:38Z

    On Mon, Aug 22, 2022 at 7:11 PM Erik Rijkers <er@xs4all.nl> wrote:
>
> Op 22-08-2022 om 10:27 schreef Peter Smith:
> >
> > PSA new set of v2* patches.
>
> Hi,
>
> In the second file a small typo, I think:
>
> "enclosed by parenthesis"  should be
> "enclosed by parentheses"
>

Thanks for your feedback.

Fixed in the v3* patches [1].

------
[1] https://www.postgresql.org/message-id/CAHut%2BPtHgQbFs9DDeOoqqQLZmMBD8FQPK2WOXJpR1nyDQy8AGA%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  245. Re: Column Filtering in Logical Replication

    vignesh C <vignesh21@gmail.com> — 2022-08-25T09:38:39Z

    On Tue, Aug 23, 2022 at 7:52 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Aug 22, 2022 at 9:25 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> ...
>
> > Few comments:
> > 1) I felt no expressions are allowed in case of column filters. Only
> > column names can be specified. The second part of the sentence
> > confuses what is allowed and what is not allowed. Won't it be better
> > to remove the second sentence and mention that only column names can
> > be specified.
> > +   <para>
> > +    Column list can contain only simple column references. Complex
> > +    expressions, function calls etc. are not allowed.
> > +   </para>
> >
>
> This wording was lifted verbatim from the commit message [1]. But I
> see your point that it just seems to be overcomplicating a simple
> rule. Modified as suggested.
>
> > 2) tablename should be table name.
> > +   <para>
> > +    A column list is specified per table following the tablename, and
> > enclosed by
> > +    parenthesis. See <xref linkend="sql-createpublication"/> for details.
> > +   </para>
> >
> > We have used table name in the same page in other instances like:
> > a) The row filter is defined per table. Use a WHERE clause after the
> > table name for each published table that requires data to be filtered
> > out. The WHERE clause must be enclosed by parentheses.
> > b) The tables are matched between the publisher and the subscriber
> > using the fully qualified table name.
> >
>
> Fixed as suggested.
>
> > 3) One small whitespace issue:
> > git am v2-0001-Column-List-replica-identity-rules.patch
> > Applying: Column List replica identity rules.
> > .git/rebase-apply/patch:30: trailing whitespace.
> >    if the publication publishes only <command>INSERT</command> operations.
> > warning: 1 line adds whitespace errors.
> >
>
> Fixed.
>
> ~~~
>
> PSA the v3* patch set.

Thanks for the updated patch.
Few comments:
1) We can shuffle the columns in publisher table and subscriber to
show that the order of the column does not matter
+   <para>
+    Create a publication <literal>p1</literal>. A column list is defined for
+    table <literal>t1</literal> to reduce the number of columns that will be
+    replicated.
+<programlisting>
+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, a, b, c);
+CREATE PUBLICATION
+test_pub=#
+</programlisting></para>

2) We can try to keep the line content to less than 80 chars
+   <para>
+    A column list is specified per table following the tablename, and
enclosed by
+    parenthesis. See <xref linkend="sql-createpublication"/> for details.
+   </para>

3) tablename should be table name like it is used in other places
+   <para>
+    A column list is specified per table following the tablename, and
enclosed by
+    parenthesis. See <xref linkend="sql-createpublication"/> for details.
+   </para>

4a) In the below, you could include mentioning "Only the column list
data of publication <literal>p1</literal> are replicated."
+    <para>
+     Insert some rows to table <literal>t1</literal>.
+<programlisting>
+test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1');
+INSERT 0 1

4b) In the above, we could mention that the insert should be done on
the "publisher side" as the previous statements are executed on the
subscriber side.

Regards,
Vignesh

  246. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-08-26T02:02:57Z

    On Thu, Aug 25, 2022 at 7:38 PM vignesh C <vignesh21@gmail.com> wrote:
>
...
> > PSA the v3* patch set.
>
> Thanks for the updated patch.
> Few comments:
> 1) We can shuffle the columns in publisher table and subscriber to
> show that the order of the column does not matter
> +   <para>
> +    Create a publication <literal>p1</literal>. A column list is defined for
> +    table <literal>t1</literal> to reduce the number of columns that will be
> +    replicated.
> +<programlisting>
> +test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, a, b, c);
> +CREATE PUBLICATION
> +test_pub=#
> +</programlisting></para>
>

OK. I made the following changes to the example.
- now the subscriber table defines cols in a different order than that
of the publisher table
- now the publisher column list defines col names in a different order
than that of the table
- now the column list avoids using only adjacent column names

> 2) We can try to keep the line content to less than 80 chars
> +   <para>
> +    A column list is specified per table following the tablename, and
> enclosed by
> +    parenthesis. See <xref linkend="sql-createpublication"/> for details.
> +   </para>
>

OK. Modified to use < 80 chars

> 3) tablename should be table name like it is used in other places
> +   <para>
> +    A column list is specified per table following the tablename, and
> enclosed by
> +    parenthesis. See <xref linkend="sql-createpublication"/> for details.
> +   </para>
>

Sorry, I don't see this problem. AFAIK this same issue was already
fixed in the v3* patches.  Notice in the cited fragment that
'parenthesis' is misspelt but that was also fixed in v3. Maybe you are
looking at an old patch file (??)

> 4a) In the below, you could include mentioning "Only the column list
> data of publication <literal>p1</literal> are replicated."
> +    <para>
> +     Insert some rows to table <literal>t1</literal>.
> +<programlisting>
> +test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1');
> +INSERT 0 1
>

OK. Modified to say this.

> 4b) In the above, we could mention that the insert should be done on
> the "publisher side" as the previous statements are executed on the
> subscriber side.

OK. Modified to say this.

~~~

Thanks for the feedback.

PSA patch set v4* where all of the above comments are now addressed.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  247. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-01T09:53:30Z

    On Fri, Aug 26, 2022 at 7:33 AM Peter Smith <smithpb2250@gmail.com> wrote:
>

Few comments on both the patches:
v4-0001*
=========
1.
Furthermore, if the table uses
+   <literal>REPLICA IDENTITY FULL</literal>, specifying a column list is not
+   allowed (it will cause publication errors for <command>UPDATE</command> or
+   <command>DELETE</command> operations).

This line sounds a bit unclear to me. From this like it appears that
the following operation is not allowed:

postgres=# create table t1(c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# Alter Table t1 replica identity full;
ALTER TABLE
postgres=# create publication pub1 for table t1(c1);
CREATE PUBLICATION

However, what is not allowed is the following:
postgres=# delete from t1;
ERROR:  cannot delete from table "t1"
DETAIL:  Column list used by the publication does not cover the
replica identity.

I am not sure if we really need this line but if so then please try to
make it more clear. I think the similar text is present in 0002 patch
which should be modified accordingly.

V4-0002*
=========
2.
However, if a
+   <firstterm>column list</firstterm> is specified then only the columns named
+   in the list will be replicated. This means the subscriber-side table only
+   needs to have those columns named by the column list. A user might choose to
+   use column lists for behavioral, security or performance reasons.
+  </para>
+
+  <sect2 id="logical-replication-col-list-rules">
+   <title>Column List Rules</title>
+
+   <para>
+    A column list is specified per table following the table name, and enclosed
+    by parentheses. See <xref linkend="sql-createpublication"/> for details.
+   </para>
+
+   <para>
+    When a column list is specified, only the named columns are replicated.
+    The list order is not important.

It seems like "When a column list is specified, only the named columns
are replicated." is almost a duplicate of the line in the first para.
So, I think we can remove it. And if we do so then the second line
could be changed to something like: "While specifying column list, the
order of columns is not important."

3. It seems information about initial table synchronization is
missing. We copy only columns specified in the column list. Also, it
would be good to add a Note similar to Row Filter to indicate that
this list won't be used by pre-15 publishers.

-- 
With Regards,
Amit Kapila.

  248. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-09-02T03:15:33Z

    On Thu, Sep 1, 2022 at 7:53 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Aug 26, 2022 at 7:33 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
>
> Few comments on both the patches:
> v4-0001*
> =========
> 1.
> Furthermore, if the table uses
> +   <literal>REPLICA IDENTITY FULL</literal>, specifying a column list is not
> +   allowed (it will cause publication errors for <command>UPDATE</command> or
> +   <command>DELETE</command> operations).
>
> This line sounds a bit unclear to me. From this like it appears that
> the following operation is not allowed:
>
> postgres=# create table t1(c1 int, c2 int, c3 int);
> CREATE TABLE
> postgres=# Alter Table t1 replica identity full;
> ALTER TABLE
> postgres=# create publication pub1 for table t1(c1);
> CREATE PUBLICATION
>
> However, what is not allowed is the following:
> postgres=# delete from t1;
> ERROR:  cannot delete from table "t1"
> DETAIL:  Column list used by the publication does not cover the
> replica identity.
>
> I am not sure if we really need this line but if so then please try to
> make it more clear. I think the similar text is present in 0002 patch
> which should be modified accordingly.
>

The "Furthermore…" sentence came from the commit message [1]. But I
agree it seems redundant/ambiguous, so I have removed it (and removed
the same in patch 0002).


> V4-0002*
> =========
> 2.
> However, if a
> +   <firstterm>column list</firstterm> is specified then only the columns named
> +   in the list will be replicated. This means the subscriber-side table only
> +   needs to have those columns named by the column list. A user might choose to
> +   use column lists for behavioral, security or performance reasons.
> +  </para>
> +
> +  <sect2 id="logical-replication-col-list-rules">
> +   <title>Column List Rules</title>
> +
> +   <para>
> +    A column list is specified per table following the table name, and enclosed
> +    by parentheses. See <xref linkend="sql-createpublication"/> for details.
> +   </para>
> +
> +   <para>
> +    When a column list is specified, only the named columns are replicated.
> +    The list order is not important.
>
> It seems like "When a column list is specified, only the named columns
> are replicated." is almost a duplicate of the line in the first para.
> So, I think we can remove it. And if we do so then the second line
> could be changed to something like: "While specifying column list, the
> order of columns is not important."
>

Modified as suggested.

> 3. It seems information about initial table synchronization is
> missing. We copy only columns specified in the column list. Also, it
> would be good to add a Note similar to Row Filter to indicate that
> this list won't be used by pre-15 publishers.
>

Done as suggested. Added a new "Initial Data Synchronization" section
with content similar to that of the Row Filters section.

~~~

Thanks for your review comments.

PSA v5* patches where all the above have been addressed.

------
[1] https://github.com/postgres/postgres/commit/923def9a533a7d986acfb524139d8b9e5466d0a5

Kind Regards,
Peter Smith.
Fujitsu Australia

  249. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-02T13:40:02Z

    On Fri, Sep 2, 2022 at 8:45 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Thu, Sep 1, 2022 at 7:53 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, Aug 26, 2022 at 7:33 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> >
> > Few comments on both the patches:
> > v4-0001*
> > =========
> > 1.
> > Furthermore, if the table uses
> > +   <literal>REPLICA IDENTITY FULL</literal>, specifying a column list is not
> > +   allowed (it will cause publication errors for <command>UPDATE</command> or
> > +   <command>DELETE</command> operations).
> >
> > This line sounds a bit unclear to me. From this like it appears that
> > the following operation is not allowed:
> >
> > postgres=# create table t1(c1 int, c2 int, c3 int);
> > CREATE TABLE
> > postgres=# Alter Table t1 replica identity full;
> > ALTER TABLE
> > postgres=# create publication pub1 for table t1(c1);
> > CREATE PUBLICATION
> >
> > However, what is not allowed is the following:
> > postgres=# delete from t1;
> > ERROR:  cannot delete from table "t1"
> > DETAIL:  Column list used by the publication does not cover the
> > replica identity.
> >
> > I am not sure if we really need this line but if so then please try to
> > make it more clear. I think the similar text is present in 0002 patch
> > which should be modified accordingly.
> >
>
> The "Furthermore…" sentence came from the commit message [1]. But I
> agree it seems redundant/ambiguous, so I have removed it (and removed
> the same in patch 0002).
>

Thanks, pushed your first patch.

-- 
With Regards,
Amit Kapila.

  250. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-09-05T00:28:11Z

    On Fri, Sep 2, 2022 at 11:40 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Sep 2, 2022 at 8:45 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Thu, Sep 1, 2022 at 7:53 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Fri, Aug 26, 2022 at 7:33 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > >
> > > Few comments on both the patches:
> > > v4-0001*
> > > =========
> > > 1.
> > > Furthermore, if the table uses
> > > +   <literal>REPLICA IDENTITY FULL</literal>, specifying a column list is not
> > > +   allowed (it will cause publication errors for <command>UPDATE</command> or
> > > +   <command>DELETE</command> operations).
> > >
> > > This line sounds a bit unclear to me. From this like it appears that
> > > the following operation is not allowed:
> > >
> > > postgres=# create table t1(c1 int, c2 int, c3 int);
> > > CREATE TABLE
> > > postgres=# Alter Table t1 replica identity full;
> > > ALTER TABLE
> > > postgres=# create publication pub1 for table t1(c1);
> > > CREATE PUBLICATION
> > >
> > > However, what is not allowed is the following:
> > > postgres=# delete from t1;
> > > ERROR:  cannot delete from table "t1"
> > > DETAIL:  Column list used by the publication does not cover the
> > > replica identity.
> > >
> > > I am not sure if we really need this line but if so then please try to
> > > make it more clear. I think the similar text is present in 0002 patch
> > > which should be modified accordingly.
> > >
> >
> > The "Furthermore…" sentence came from the commit message [1]. But I
> > agree it seems redundant/ambiguous, so I have removed it (and removed
> > the same in patch 0002).
> >
>
> Thanks, pushed your first patch.
>

Thanks for the push.

I have rebased the remaining patch (v6-0001 is the same as v5-0002)

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  251. RE: Column Filtering in Logical Replication

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-09-05T03:42:40Z

    On Mon, Sep 5, 2022 8:28 AM Peter Smith <smithpb2250@gmail.com> wrote:
> 
> I have rebased the remaining patch (v6-0001 is the same as v5-0002)
> 

Thanks for updating the patch. Here are some comments.

1.
+     the <xref linkend="sql-alterpublication"/> will be successful but later
+     the WalSender on the publisher, or the subscriber may throw an error. In
+     this scenario, the user needs to recreate the subscription after adjusting

Should "WalSender" be changed to "walsender"? I saw "walsender" is used in other
places in the documentation.

2.
+test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id));
+CREATE TABLE
+test_pub=#

+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, b, a, d);
+CREATE PUBLICATION
+test_pub=#

I think the redundant "test_pub=#" can be removed.


Besides, I tested the examples in the patch, there's no problem.

Regards,
Shi yu

  252. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-09-05T10:16:09Z

    On Mon, Sep 5, 2022 at 1:42 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Mon, Sep 5, 2022 8:28 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > I have rebased the remaining patch (v6-0001 is the same as v5-0002)
> >
>
> Thanks for updating the patch. Here are some comments.
>
> 1.
> +     the <xref linkend="sql-alterpublication"/> will be successful but later
> +     the WalSender on the publisher, or the subscriber may throw an error. In
> +     this scenario, the user needs to recreate the subscription after adjusting
>
> Should "WalSender" be changed to "walsender"? I saw "walsender" is used in other
> places in the documentation.

Modified.

>
> 2.
> +test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id));
> +CREATE TABLE
> +test_pub=#
>
> +test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, b, a, d);
> +CREATE PUBLICATION
> +test_pub=#
>
> I think the redundant "test_pub=#" can be removed.
>

Modified.

>
> Besides, I tested the examples in the patch, there's no problem.
>

Thanks for the review comments, and testing.

I made both fixes as suggested.

PSA v7.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  253. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-05T10:46:10Z

    On Mon, Sep 5, 2022 at 3:46 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
>
> PSA v7.
>

For example, if additional columns are added to the table, then
+    (after a <literal>REFRESH PUBLICATION</literal>) if there was a column list
+    only those named columns will continue to be replicated.

This looks a bit unclear to me w.r.t the refresh publication step. Why
exactly you have used refresh publication in the above para? It is
used to add new tables if any added to the publication, so not clear
to me how it helps in this case. If that is not required then we can
change it to: "For example, if additional columns are added to the
table then only those named columns mentioned in the column list will
continue to be replicated."

-- 
With Regards,
Amit Kapila.

  254. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-09-05T23:37:45Z

    On Mon, Sep 5, 2022 at 8:46 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Sep 5, 2022 at 3:46 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> >
> > PSA v7.
> >
>
> For example, if additional columns are added to the table, then
> +    (after a <literal>REFRESH PUBLICATION</literal>) if there was a column list
> +    only those named columns will continue to be replicated.
>
> This looks a bit unclear to me w.r.t the refresh publication step. Why
> exactly you have used refresh publication in the above para? It is
> used to add new tables if any added to the publication, so not clear
> to me how it helps in this case. If that is not required then we can
> change it to: "For example, if additional columns are added to the
> table then only those named columns mentioned in the column list will
> continue to be replicated."
>

You are right - that REFRESH PUBLICATION was not necessary for this
example. The patch is modified to use your suggested text.

PSA v8

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  255. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-06T08:45:22Z

    On Tue, Sep 6, 2022 at 5:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> You are right - that REFRESH PUBLICATION was not necessary for this
> example. The patch is modified to use your suggested text.
>
> PSA v8
>

LGTM. I'll push this once the tag appears for v15.

-- 
With Regards,
Amit Kapila.

  256. Re: Column Filtering in Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-07T10:48:48Z

    On Tue, Sep 6, 2022 at 2:15 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Sep 6, 2022 at 5:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > You are right - that REFRESH PUBLICATION was not necessary for this
> > example. The patch is modified to use your suggested text.
> >
> > PSA v8
> >
>
> LGTM. I'll push this once the tag appears for v15.
>

Pushed!

-- 
With Regards,
Amit Kapila.

  257. Re: Column Filtering in Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2022-09-07T22:31:52Z

    On Wed, Sep 7, 2022 at 8:49 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Sep 6, 2022 at 2:15 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Sep 6, 2022 at 5:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > You are right - that REFRESH PUBLICATION was not necessary for this
> > > example. The patch is modified to use your suggested text.
> > >
> > > PSA v8
> > >
> >
> > LGTM. I'll push this once the tag appears for v15.
> >
>
> Pushed!

Thanks for pushing.

------
Kind Regards,
Peter Smith.
Fujitsu Australia.