Horton
Menu

Re: Column Filtering in Logical Replication

Alvaro Herrera <alvherre@alvh.no-ip.org>

View thread
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>, Rahila Syed <rahilasyed90@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-12-27T18:31:46Z
Lists: pgsql-hackers 
On 2021-Dec-27, Tom Lane wrote:

> Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> > Determining that an array has a NULL element seems convoluted.  I ended
> > up with this query, where comparing the result of array_positions() with
> > an empty array does that.  If anybody knows of a simpler way, or any
> > situations in which this fails, I'm all ears.
> 
> Maybe better to rethink why we allow elements of prattrs to be null?

What I'm doing is an unnest of all arrays and then aggregating them
back into a single array.  If one array is null, the resulting aggregate
contains a null element.

Hmm, maybe I can in parallel do a bool_or() aggregate of "array is null" to
avoid that.  ... ah yes, that works:

with published_cols as (
        select pg_catalog.bool_or(pr.prattrs is null) as all_columns,
                pg_catalog.array_agg(distinct unnest order by unnest) AS attrs
        from pg_catalog.pg_publication p join
                pg_catalog.pg_publication_rel pr on (p.oid = pr.prpubid) left join
                unnest(prattrs) on (true)
        where prrelid = :table and p.pubname in ('pub1', 'pub2')
)
SELECT a.attname,
       a.atttypid,
       a.attnum = ANY(i.indkey)
  FROM pg_catalog.pg_attribute a
  LEFT JOIN pg_catalog.pg_index i
       ON (i.indexrelid = pg_get_replica_identity_index(:table)),
     published_cols
 WHERE a.attnum > 0::pg_catalog.int2
   AND NOT a.attisdropped and a.attgenerated = ''
   AND a.attrelid = :table
   AND (all_columns OR attnum = ANY(published_cols.attrs))
 ORDER BY a.attnum ;

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Explain about Column List feature.

  2. Doc: fix column list vs. replica identity rules.

  3. Prohibit combining publications with different column lists.

  4. Fix the check to limit sync workers.

  5. Wait for subscription to sync in t/031_column_list.sql

  6. Move prattrs to the pg_publication_rel section in docs

  7. Allow specifying column lists for logical replication

  8. Fix row filters with multiple publications

  9. Fix publish_as_relid with multiple publications

  10. Add some additional tests for row filters in logical replication.

  11. Add index on pg_publication_rel.prpubid

  12. Avoid using DefElemAction in AlterPublicationStmt

  13. Small cleanups related to PUBLICATION framework code

  14. Add PublicationTable and PublicationRelInfo structs

  15. Fix various concurrency issues in logical replication worker launching