Horton
Menu

Re: Column Filtering in Logical Replication

Tomas Vondra <tomas.vondra@enterprisedb.com>

View thread
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Peter Eisentraut <peter.eisentraut@enterprisedb.com>, Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: Justin Pryzby <pryzby@telsasoft.com>, Amit Kapila <amit.kapila16@gmail.com>, "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, Rahila Syed <rahilasyed90@gmail.com>, Peter Smith <smithpb2250@gmail.com>, pgsql-hackers@postgresql.org
Date: 2022-02-15T23:33:37Z
Lists: pgsql-hackers

Attachments

Hi,

Here's an updated version of the patch, rebased to current master. Parts 
0002 and 0003 include various improvements based on review by me and 
another one by Peter Smith [1].

Part 0003 reworks and significantly extends the TAP test, to exercise 
various cases related to changes of replica identity etc. discussed in 
this thread. Some of the tests however still fail, because the behavior 
was not updated - I'll work on that once we agree what the expected 
behavior is.

1) partitioning with pubviaroot=true

The main set of failures is related to partitions with different replica 
identities and (pubviaroot=true), some of which may be mismatching the 
column list. There are multiple such test cases, depending on how the 
inconsistency is introduced - it may be there from the beginning, the 
column filter may be modified after adding the partitioned table to the 
publication, etc.

I think the expected behavior is to prohibit such cases from happening, 
by cross-checking the column filter when adding the partitioned table to 
publication, attaching a partition or changing a column filter.


2) merging multiple column filters

When the table has multiple column filters (in different publications), 
we need to merge them. Which works, except that FOR ALL TABLES [IN 
SCHEMA] needs to be handled as "has no column filter" (and replicates 
everything).


3) partitioning with pubivaroot=false

When a partitioned table is added with (pubviaroot=false), it should not 
be subject to column filter on the parent relation, which is the same 
behavior used by the row filtering patch.


regards


[1] 
https://www.postgresql.org/message-id/CAHut%2BPtc7Rh187eQKrxdUmUNWyfxz7OkhYAX%3DAW411Qwxya0LQ%40mail.gmail.com

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Explain about Column List feature.

  2. Doc: fix column list vs. replica identity rules.

  3. Prohibit combining publications with different column lists.

  4. Fix the check to limit sync workers.

  5. Wait for subscription to sync in t/031_column_list.sql

  6. Move prattrs to the pg_publication_rel section in docs

  7. Allow specifying column lists for logical replication

  8. Fix row filters with multiple publications

  9. Fix publish_as_relid with multiple publications

  10. Add some additional tests for row filters in logical replication.

  11. Add index on pg_publication_rel.prpubid

  12. Avoid using DefElemAction in AlterPublicationStmt

  13. Small cleanups related to PUBLICATION framework code

  14. Add PublicationTable and PublicationRelInfo structs

  15. Fix various concurrency issues in logical replication worker launching