Horton

Re: Data is copied twice when specifying both child and parent table in publication

Jacob Champion <jchampion@timescale.com>

View thread View on postgr.es
From: Jacob Champion <jchampion@timescale.com>
To: "shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>, Amit Kapila <amit.kapila16@gmail.com>
Cc: "wangw.fnst@fujitsu.com" <wangw.fnst@fujitsu.com>, Andres Freund <andres@anarazel.de>, vignesh C <vignesh21@gmail.com>, Peter Smith <smithpb2250@gmail.com>, "Takamichi Osumi (Fujitsu)" <osumi.takamichi@fujitsu.com>, "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, Amit Langote <amitlangote09@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>, Dilip Kumar <dilipbalaut@gmail.com>, Greg Nancarrow <gregn4422@gmail.com>
Date: 2023-03-31T23:05:37Z
Lists: pgsql-hackers 
On 3/31/23 03:04, shiy.fnst@fujitsu.com wrote:
> I noticed that a similar problem has been discussed in this thread, see [1] [2]
> [3] [4].

Ah, thank you. I didn't go far back enough in the thread...

> It seems complicated to fix it if we want to automatically skip tables
> that have been synchronized previously by code

I agree, this is looking very complex. I need to read through the
examples you sent more closely.

> and this may overkill in some
> cases (e.g. The target table in subscriber is not a partitioned table, and the
> user want to synchronize all data in the partitioned table from the publisher).

Hm. It seems like the setup process doesn't really capture the user's
intent. There are just so many things that they could be theoretically
trying to do.

> Besides, it seems not a common case. So I'm not sure we should fix it. Maybe we
> can just add some documentation for it as Peter mentioned.

I think we should absolutely document the pitfalls here. (I'm still
trying to figure out what they are, though, so I don't have any concrete
suggestions yet...)

Thanks!
--Jacob

Commits

  1. Avoid syncing data twice for the 'publish_via_partition_root' option.

  2. Fix partition table's REPLICA IDENTITY checking on the subscriber.

  3. Prohibit combining publications with different column lists.

  4. Fix double publish of child table's data.

  5. Support adding partitioned tables to publication