Horton
Menu

Re: Column Filtering in Logical Replication

Peter Eisentraut <peter.eisentraut@enterprisedb.com>

View thread
From: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
To: Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-12-10T13:08:52Z
Lists: pgsql-hackers

Attachments

On 02.12.21 15:23, Alvaro Herrera wrote:
>> I took the latest posted patch, rebased on current sources, fixed the
>> conflicts, and pgindented.  No further changes.  Here's the result.  All
>> tests are passing for me.  Some review comments that were posted have
>> not been addressed yet; I'll look into that soon.
> 
> In v7 I have reinstated the test file and fixed the silly problem that
> caused it to fail (probably a mistake of mine while rebasing).

I looked through this a bit.  You had said that you are still going to 
integrate past review comments, so I didn't look to deeply before you 
get to that.

Attached are a few fixup patches that you could integrate.

There was no documentation, so I wrote a bit (patch 0001).  It only 
touches the CREATE PUBLICATION and ALTER PUBLICATION pages at the 
moment.  There was no mention in the Logical Replication chapter that 
warranted updating.  Perhaps we should revisit that chapter at the end 
of the release cycle.

DDL tests should be done in src/test/regress/sql/publication.sql rather 
than through TAP tests, to keep it simpler.  I have added a few that I 
came up with (patch 0002).  Note the FIXME marker that it does not 
recognize if the listed columns don't exist.  I removed a now redundant 
test from the TAP test file.  The other error condition test in the TAP 
test file ('publication relation test_part removed') I didn't 
understand: test_part was added with columns (a, b), so why would 
dropping column b remove the whole entry?  Maybe I missed something, or 
this could be explained better.

I was curious what happens when you have different publications with 
different column lists, so I wrote a test for that (patch 0003).  It 
turns out it works, so there is nothing to do, but perhaps the test is 
useful to keep.

The test file 021_column_filter.pl should be renamed to an unused number 
(would be 027 currently).  Also, it contains references to "TRUNCATE", 
where it was presumably copied from.

On the implementation side, I think the added catalog column 
pg_publication_rel.prattrs should be an int2 array, not a text array. 
That would also fix the above problem.  If you have to look up the 
columns at DDL time, then you will notice when they don't exist.

Finally, I suggest not naming this feature "column filter".  I think 
this name arose because of the analogy with the "row filter" feature 
also being developed.  But a filter is normally a dynamic data-driven 
action, which this is not.  Golden Gate calls it in their documentation 
"Selecting Columns", or we could just call it "column list".

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Explain about Column List feature.

  2. Doc: fix column list vs. replica identity rules.

  3. Prohibit combining publications with different column lists.

  4. Fix the check to limit sync workers.

  5. Wait for subscription to sync in t/031_column_list.sql

  6. Move prattrs to the pg_publication_rel section in docs

  7. Allow specifying column lists for logical replication

  8. Fix row filters with multiple publications

  9. Fix publish_as_relid with multiple publications

  10. Add some additional tests for row filters in logical replication.

  11. Add index on pg_publication_rel.prpubid

  12. Avoid using DefElemAction in AlterPublicationStmt

  13. Small cleanups related to PUBLICATION framework code

  14. Add PublicationTable and PublicationRelInfo structs

  15. Fix various concurrency issues in logical replication worker launching