Re: Support logical replication of DDLs
Zheng Li <zhengli10@gmail.com>
From: Zheng Li <zhengli10@gmail.com>
To: li jie <ggysxcq@gmail.com>
Cc: vignesh C <vignesh21@gmail.com>, Peter Smith <smithpb2250@gmail.com>, Ajin Cherian <itsajin@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, Alvaro Herrera <alvherre@alvh.no-ip.org>,
"houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, Amit Kapila <amit.kapila16@gmail.com>,
Masahiko Sawada <sawada.mshk@gmail.com>, Japin Li <japinli@hotmail.com>, rajesh singarapu <rajesh.rs0541@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2022-11-28T04:47:39Z
Lists: pgsql-hackers
Commits
Same data as JSON:
GET /api/v1/messages/:b64id/commits
the thread's linked commits as JSON, with link sources.
API reference →
-
Add a run_as_owner option to subscriptions.
- 482675987bcd 16.0 cited
-
Refactor pgoutput_change().
- da324d6cd45b 16.0 cited
-
Print the correct aliases for DML target tables in ruleutils.
- df931e9ab35b 11.20 landed
- c8a5f1685fb7 15.3 landed
- 4efb4f0d4878 13.11 landed
- 3dd287c14fac 12.15 landed
- 393430f57544 16.0 landed
- 14345f3c6a7b 14.8 landed
-
Fix object identity string for transforms
- 9a312562314a 16.0 landed
-
Add grantable MAINTAIN privilege and pg_maintain role.
- 60684dd834a2 16.0 cited
-
Get rid of recursion-marker values in enum AlterTableType
- 840ff5f451cd 16.0 cited
-
Release cache tuple when no longer needed
- ed0fbc8e5ac9 15.0 cited
-
Empty search_path in logical replication apply worker and walsender.
- 11da97024abb 14.0 cited
-
Refactor format_type APIs to be more modular
- a26116c6cbf4 11.0 cited
-
Use wrappers of PG_DETOAST_DATUM_PACKED() more.
- 3a0d473192b2 10.0 cited
Attachments
- v40-0002-Support-DDL-replication.patch (application/octet-stream) patch v40-0002
- v40-0005-Do-not-generate-WAL-log-for-non-top-level-DDL-comman.patch (application/octet-stream) patch v40-0005
- v40-0004-Test-cases-for-DDL-replication.patch (application/octet-stream) patch v40-0004
- v40-0001-Functions-to-deparse-DDL-commands.patch (application/octet-stream) patch v40-0001
- v40-0003-Support-CREATE-TABLE-AS-SELECT-INTO.patch (application/octet-stream) patch v40-0003
> > >
> > > 1. CREATE TABLE LIKE
> > > I found that this case may be repication incorrectly.
> > > You can run the following SQL statement:
> > > ```
> > > CREATE TABLE ctlt1 (a text CHECK (length(a) > 2) PRIMARY KEY, b text);
> > > ALTER TABLE ctlt1 ALTER COLUMN a SET STORAGE MAIN;
> > > ALTER TABLE ctlt1 ALTER COLUMN b SET STORAGE EXTERNAL;
> > > CREATE TABLE ctlt1_like (LIKE ctlt1 INCLUDING ALL);
> > > ```
> > > The ctlt1_like table will not be able to correct the replication.
> > > I think this is because create table like statement is captured by
> > > the event trigger to a create table statement and multiple alter table statements.
> > > There are some overlaps between them, and an error is reported when downstream replication occurs.
> >
> > I looked into this case. The root cause is the statement
> >
> > CREATE TABLE ctlt1_like (LIKE ctlt1 INCLUDING ALL);
> >
> > is executed internally using 3 DDLs:
> > 1. CREATE TABLE ctlt1_like (LIKE ctlt1 INCLUDING ALL); --The top level command
> > 2. ALTER TABLE ctlt1_like ADD CONSTRAINT ctlt1_a_check CHECK
> > (length(a) > 2); --The first subcommand
> > 3. CREATE UNIQUE INDEX ctlt1_like_pkey on ctlt1_like (a); --The second
> > subcommand that creates the primary key index
> >
> > All three commands are captured by the event trigger. The first and
> > second command ends up getting deparsed, WAL-logged and
> > replayed on the subscriber. The replay of the ALTER TABLE command
> > causes a duplicate constraint error. The problem is that
> > while subcommands are captured by event triggers by default, they
> > don't need to be deparsed and WAL-logged for DDL replication.
> > To do that we can pass the isCompleteQuery variable in
> > ProcessUtilitySlow to EventTriggerCollectSimpleCommand() and
> > EventTriggerAlterTableEnd() and make this information available in
> > CollectedCommand so that any subcommands can be skipped.
>
> Attaching the proposed fix in
> v40-0005-Do-not-generate-WAL-log-for-non-top-level-DDL-comman.patch.
> This patch adds a new boolean field isTopLevelCommand to
> CollectedCommand so that non-top level command
> can be skipped in the DDL replication event trigger functions. The
> patch also makes the information available by
> passing the isTopLevel variable in ProcessUtilitySlow to several
> EventTriggerCollect functions such as
> EventTriggerCollectSimpleCommand and EventTriggerAlterTableStart.
Patch v40-0005-Do-not-generate-WAL-log-for-non-top-level-DDL-comman.patch
broke the following test case:
CREATE TABLE product (id int PRIMARY KEY, name text);
CREATE TABLE orders (order_id int PRIMARY KEY, product_id int
REFERENCES product (id));
Because forein key constraint was not deparsed as part of CREATE TABLE
but rather replicated as a non-top-level command (which we
no longer replicate in patch v40-0005), fixed this in the attached patch:
diff --git a/src/backend/commands/ddl_deparse.c
b/src/backend/commands/ddl_deparse.c
index 89f33d451c..d9bb3aab8b 100755
--- a/src/backend/commands/ddl_deparse.c
+++ b/src/backend/commands/ddl_deparse.c
@@ -1087,8 +1087,6 @@ objtree_to_jsonb_rec(ObjTree *tree,
JsonbParseState *state)
* the given elements list. The updated list is returned.
*
* This works for typed tables, regular tables, and domains.
- *
- * Note that CONSTRAINT_FOREIGN constraints are always ignored.
*/
static List *
obtainConstraints(List *elements, Oid relationId, Oid domainId)
@@ -1146,7 +1144,8 @@ obtainConstraints(List *elements, Oid
relationId, Oid domainId)
contype = "check";
break;
case CONSTRAINT_FOREIGN:
- continue; /* not here */
+ contype = "foreign key";
+ break;
case CONSTRAINT_PRIMARY:
contype = "primary key";
break;
Regards,
Zheng