Data is copied twice when specifying both child and parent table in publication
Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>
From: "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>
To: "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>
Cc: Greg Nancarrow <gregn4422@gmail.com>
Date: 2021-10-15T11:22:50Z
Lists: pgsql-hackers
Attachments
- 0001-fix-double-publish.patch (application/octet-stream) patch 0001
Hi, In another logical replication related thread[1], my colleague Greg found that if publish_via_partition_root is true, then the child table's data will be copied twice when adding both child and parent table to the publication. Example: ----- Pub: create table tbl1 (a int) partition by range (a); create table tbl1_part1 partition of tbl1 for values from (1) to (10); create table tbl1_part2 partition of tbl1 for values from (10) to (20); create publication pub for table tbl1, tbl1_part1 with (publish_via_partition_root=on); insert into tbl1_part1 values(1); Sub: create table tbl1 (a int) partition by range (a); create table tbl1_part1 partition of tbl1 for values from (1) to (10); create table tbl1_part2 partition of tbl1 for values from (10) to (20); create subscription sub CONNECTION 'dbname=postgres port=10000' publication pub; -- data is copied twice select * from tbl1_part1; a --- 1 1 ----- The reason is that the subscriber will fetch the table list from publisher using the following sql[2] and the subscriber will execute table synchronization for each table in the query results in this case. But tbl1_part1 is a partition of tbl1, so the data of tbl1_part1 was copied twice. [2] select * from pg_publication_tables; pubname | schemaname | tablename ---------+------------+------------ pub | public | tbl1 pub | public | tbl1_part1 IMO, it looks like a bug and it's more natural to only execute the table synchronization for the parent table in the above case. Because as the document said: if publish_via_partition_root is true, "changes in a partitioned table (or on its partitions) contained in the publication will be published using the identity and schema of the partitioned table rather than that of the individual partitions that are actually changed;" To fix it, I think we should fix function GetPublicationRelations which generate data for the view pg_publication_tables and make it only show the parent table if publish_via_partition_root is true. And for other future feature like schema level publication, we can also follow this to exclude partitions if their parent is specified by FOR TABLE in the same publication. Attach a patch to fix it. Thoughts ? [1] https://www.postgresql.org/message-id/CAJcOf-eBhDUT2J5zs8Z0qEMiZUdhinX%2BbuGX3GN4V83fPnZV3Q%40mail.gmail.com Best regards, Hou zhijie
Commits
-
Avoid syncing data twice for the 'publish_via_partition_root' option.
- 062a84442424 16.0 landed
-
Fix partition table's REPLICA IDENTITY checking on the subscriber.
- 52d5ea9adb01 14.5 cited
-
Prohibit combining publications with different column lists.
- fd0b9dcebda7 15.0 cited
-
Fix double publish of child table's data.
- 3f06c00cf6dc 13.6 landed
- 614b77d65a38 14.2 landed
- 5e97905a2c76 15.0 landed
-
Support adding partitioned tables to publication
- 17b9e7f9fe23 13.0 cited