Horton
Menu

Re: Column Filtering in Logical Replication

Ibrar Ahmed <ibrar.ahmad@gmail.com>

View thread
From: Ibrar Ahmed <ibrar.ahmad@gmail.com>
To: Rahila Syed <rahilasyed90@gmail.com>
Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>, Peter Smith <smithpb2250@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-07-19T10:20:32Z
Lists: pgsql-hackers 
On Tue, Jul 13, 2021 at 7:44 PM Rahila Syed <rahilasyed90@gmail.com> wrote:

>
> Hi Tomas,
>
> Thank you for your comments.
>
>
>>
>> >
>> > Currently, this capability is not included in the patch. If the table on
>> > the subscriber
>> > server has lesser attributes than that on the publisher server, it
>> > throws an error at the
>> > time of CREATE SUBSCRIPTION.
>> >
>>
>> That's a bit surprising, to be honest. I do understand the patch simply
>> treats the filtered columns as "unchanged" because that's the simplest
>> way to filter the *data* of the columns. But if someone told me we can
>> "filter columns" I'd expect this to work without the columns on the
>> subscriber.
>>
>> OK, I will look into adding this.
>
>
>>
>> > However, need to carefully consider situations in which a server
>> > subscribes to multiple
>> > publications,  each publishing a different subset of columns of a
>> table.
>
> Isn't that pretty much the same situation as for multiple subscriptions
>> each with a different set of I/U/D operations? IIRC we simply merge
>> those, so why not to do the same thing here and merge the attributes?
>>
>>
> Yeah, I agree with the solution to merge the attributes, similar to how
> operations are merged. My concern was also from an implementation point
> of view, will it be a very drastic change. I now had a look at how remote
> relation
> attributes are acquired for comparison with local attributes at the
> subscriber.
> It seems that the publisher will need to send the information about the
> filtered columns
> for each publication specified during CREATE SUBSCRIPTION.
> This will be read at the subscriber side which in turn updates its cache
> accordingly.
> Currently, the subscriber expects all attributes of a published relation
> to be present.
> I will add code for this in the next version of the patch.
>
>  To nitpick, I find "Bitmapset *att_list" a bit annoying, because it's
>
> not really a list ;-)
>
>
> I will make this change with the next version
>
>
>
>>  FWIW "make check" fails for me with this version, due to segfault in
>> OpenTableLists. Apparenly there's some confusion - the code expects the
>> list to contain PublicationTable nodes, and tries to extract the
>> RangeVar from the elements. But the list actually contains RangeVar, so
>> this crashes and burns. See the attached backtrace.
>>
>>
> Thank you for the report, This is fixed in the attached version, now all
> publication
> function calls accept the PublicationTableInfo list.
>
> Thank you,
> Rahila Syed
>
>
>

The patch does not apply, and an rebase is required

Hunk #8 succeeded at 1259 (offset 99 lines).
Hunk #9 succeeded at 1360 (offset 99 lines).
1 out of 9 hunks FAILED -- saving rejects to file
src/backend/replication/pgoutput/pgoutput.c.rej
patching file src/include/catalog/pg_publication.h


Changing the status to "Waiting on Author"

-- 
Ibrar Ahmed

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Explain about Column List feature.

  2. Doc: fix column list vs. replica identity rules.

  3. Prohibit combining publications with different column lists.

  4. Fix the check to limit sync workers.

  5. Wait for subscription to sync in t/031_column_list.sql

  6. Move prattrs to the pg_publication_rel section in docs

  7. Allow specifying column lists for logical replication

  8. Fix row filters with multiple publications

  9. Fix publish_as_relid with multiple publications

  10. Add some additional tests for row filters in logical replication.

  11. Add index on pg_publication_rel.prpubid

  12. Avoid using DefElemAction in AlterPublicationStmt

  13. Small cleanups related to PUBLICATION framework code

  14. Add PublicationTable and PublicationRelInfo structs

  15. Fix various concurrency issues in logical replication worker launching