Horton
Menu

Re: Column Filtering in Logical Replication

Rahila Syed <rahilasyed90@gmail.com>

View thread
From: Rahila Syed <rahilasyed90@gmail.com>
To: Tomas Vondra <tomas.vondra@enterprisedb.com>
Cc: Peter Smith <smithpb2250@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-07-13T14:43:44Z
Lists: pgsql-hackers

Attachments

Hi Tomas,

Thank you for your comments.


>
> >
> > Currently, this capability is not included in the patch. If the table on
> > the subscriber
> > server has lesser attributes than that on the publisher server, it
> > throws an error at the
> > time of CREATE SUBSCRIPTION.
> >
>
> That's a bit surprising, to be honest. I do understand the patch simply
> treats the filtered columns as "unchanged" because that's the simplest
> way to filter the *data* of the columns. But if someone told me we can
> "filter columns" I'd expect this to work without the columns on the
> subscriber.
>
> OK, I will look into adding this.


>
> > However, need to carefully consider situations in which a server
> > subscribes to multiple
> > publications,  each publishing a different subset of columns of a table.

Isn't that pretty much the same situation as for multiple subscriptions
> each with a different set of I/U/D operations? IIRC we simply merge
> those, so why not to do the same thing here and merge the attributes?
>
>
Yeah, I agree with the solution to merge the attributes, similar to how
operations are merged. My concern was also from an implementation point
of view, will it be a very drastic change. I now had a look at how remote
relation
attributes are acquired for comparison with local attributes at the
subscriber.
It seems that the publisher will need to send the information about the
filtered columns
for each publication specified during CREATE SUBSCRIPTION.
This will be read at the subscriber side which in turn updates its cache
accordingly.
Currently, the subscriber expects all attributes of a published relation to
be present.
I will add code for this in the next version of the patch.

 To nitpick, I find "Bitmapset *att_list" a bit annoying, because it's

not really a list ;-)


I will make this change with the next version



>  FWIW "make check" fails for me with this version, due to segfault in
> OpenTableLists. Apparenly there's some confusion - the code expects the
> list to contain PublicationTable nodes, and tries to extract the
> RangeVar from the elements. But the list actually contains RangeVar, so
> this crashes and burns. See the attached backtrace.
>
>
Thank you for the report, This is fixed in the attached version, now all
publication
function calls accept the PublicationTableInfo list.

Thank you,
Rahila Syed

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Explain about Column List feature.

  2. Doc: fix column list vs. replica identity rules.

  3. Prohibit combining publications with different column lists.

  4. Fix the check to limit sync workers.

  5. Wait for subscription to sync in t/031_column_list.sql

  6. Move prattrs to the pg_publication_rel section in docs

  7. Allow specifying column lists for logical replication

  8. Fix row filters with multiple publications

  9. Fix publish_as_relid with multiple publications

  10. Add some additional tests for row filters in logical replication.

  11. Add index on pg_publication_rel.prpubid

  12. Avoid using DefElemAction in AlterPublicationStmt

  13. Small cleanups related to PUBLICATION framework code

  14. Add PublicationTable and PublicationRelInfo structs

  15. Fix various concurrency issues in logical replication worker launching