Horton
Menu

Re: Column Filtering in Logical Replication

vignesh C <vignesh21@gmail.com>

View thread
From: vignesh C <vignesh21@gmail.com>
To: Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Rahila Syed <rahilasyed90@gmail.com>, Tomas Vondra <tomas.vondra@enterprisedb.com>, Peter Smith <smithpb2250@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-09-15T12:19:19Z
Lists: pgsql-hackers

Attachments

On Wed, Sep 15, 2021 at 5:20 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2021-Sep-15, Amit Kapila wrote:
>
> > On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> > >
> > > I pushed the clerical part of this -- namely the addition of
> > > PublicationTable node and PublicationRelInfo struct.
> >
> > One point to note here is that we are developing a generic grammar for
> > publications where not only tables but other objects like schema,
> > sequences, etc. can be specified, see [1]. So, there is some overlap
> > in the grammar modifications being made by this patch and the work
> > being done in that other thread.
>
> Oh rats.  I was not aware of that thread, or indeed of the fact that
> adding multiple object types to publications was being considered.
>
> I do see that 0002 there contains gram.y changes, but AFAICS those
> changes don't allow specifying a column list for a table, so there are
> some changes needed in that patch for that either way.
>
> I agree that it's better to move forward in unison.
>
> I noticed that 0002 in that other patch uses a void * pointer in
> PublicationObjSpec that "could be either RangeVar or String", which
> strikes me as a really bad idea.  (Already discussed in some other
> thread recently, maybe this one or the row filtering one.)

I have extracted the parser code and attached it here, so that it will
be easy to go through. We wanted to support the following syntax as in
[1]:
CREATE PUBLICATION pub1 FOR
TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2,
SEQUENCE seq1,seq2, ALL SEQUENCES IN SCHEMA s3,s4;

Columns can be added to PublicationObjSpec data structure. The patch
Generic_object_type_parser_002_table_schema_publication.patch has the
changes that were used to handle the parsing. Schema and Relation both
are different objects, schema is of string type and relation is of
RangeVar type. While parsing, schema name is parsed in string format
and relation is parsed and converted to rangevar type, these objects
will be then handled accordingly during post processing. That is the
reason it used void * type which could hold both RangeVar and String.
Thoughts?

[1] - https://www.postgresql.org/message-id/877603.1629120678%40sss.pgh.pa.us

Regards,
Vignesh

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Explain about Column List feature.

  2. Doc: fix column list vs. replica identity rules.

  3. Prohibit combining publications with different column lists.

  4. Fix the check to limit sync workers.

  5. Wait for subscription to sync in t/031_column_list.sql

  6. Move prattrs to the pg_publication_rel section in docs

  7. Allow specifying column lists for logical replication

  8. Fix row filters with multiple publications

  9. Fix publish_as_relid with multiple publications

  10. Add some additional tests for row filters in logical replication.

  11. Add index on pg_publication_rel.prpubid

  12. Avoid using DefElemAction in AlterPublicationStmt

  13. Small cleanups related to PUBLICATION framework code

  14. Add PublicationTable and PublicationRelInfo structs

  15. Fix various concurrency issues in logical replication worker launching