Horton
Menu

Re: Column Filtering in Logical Replication

Alvaro Herrera <alvherre@alvh.no-ip.org>

View thread
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: Tomas Vondra <tomas.vondra@enterprisedb.com>, Amit Kapila <amit.kapila16@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, Rahila Syed <rahilasyed90@gmail.com>
Cc: Peter Smith <smithpb2250@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-12-30T00:15:08Z
Lists: pgsql-hackers

Attachments

On 2021-Dec-28, Alvaro Herrera wrote:

> There are still some XXX comments.  The one that bothers me most is the
> lack of an implementation that allows changing the column list in a
> publication without having to remove the table from the publication
> first.

OK, I made some progress on this front; I added new forms of ALTER
PUBLICATION to support it:

ALTER PUBLICATION pub1 ALTER TABLE tbl SET COLUMNS (a, b, c);
ALTER PUBLICATION pub1 ALTER TABLE tbl SET COLUMNS ALL;

(not wedded to this syntax; other suggestions welcome)

In order to implement it I changed the haphazardly chosen use of
DEFELEM actions to a new enum.  I also noticed that the division of
labor between pg_publication.c and publicationcmds.c is quite broken
(code to translate column names to numbers is in the former, should be
in the latter; some code that deals with pg_publication tuples is in the
latter, should be in the former, such as CreatePublication,
AlterPublicationOptions).

This new stuff is not yet finished.  For example I didn't refactor
handling of REPLICA IDENTITY, so the new command does not correctly
check everything, such as the REPLICA IDENTITY FULL stuff.  Also, no
tests have been added yet.  In manual tests it seems to behave as
expected.

I noticed that prattrs is inserted in user-specified order instead of
catalog order, which is innocuous but quite weird.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"No renuncies a nada. No te aferres a nada."

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Explain about Column List feature.

  2. Doc: fix column list vs. replica identity rules.

  3. Prohibit combining publications with different column lists.

  4. Fix the check to limit sync workers.

  5. Wait for subscription to sync in t/031_column_list.sql

  6. Move prattrs to the pg_publication_rel section in docs

  7. Allow specifying column lists for logical replication

  8. Fix row filters with multiple publications

  9. Fix publish_as_relid with multiple publications

  10. Add some additional tests for row filters in logical replication.

  11. Add index on pg_publication_rel.prpubid

  12. Avoid using DefElemAction in AlterPublicationStmt

  13. Small cleanups related to PUBLICATION framework code

  14. Add PublicationTable and PublicationRelInfo structs

  15. Fix various concurrency issues in logical replication worker launching