PS_v330004_restructured.txt

text/plain

Filename: PS_v330004_restructured.txt
Type: text/plain
Part: 0
Message: Re: Proposal: Conflict log history table for Logical Replication
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 9e7868487de..7e08b9c0081 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -293,6 +293,20 @@
    option of <command>CREATE SUBSCRIPTION</command> for details.
   </para>
 
+  <para>
+   Conflicts that occur during replication are, by default, logged as plain text
+   in the server log, which can make automated monitoring and analysis difficult.
+   The <command>CREATE SUBSCRIPTION</command> command provides the
+   <link linkend="sql-createsubscription-params-with-conflict-log-destination">
+   <literal>conflict_log_destination</literal></link> option to record detailed
+   conflict information in a structured, queryable format. When this parameter
+   is set to <literal>table</literal> or <literal>all</literal>, the system
+   automatically manages a dedicated <firstterm>conflict log table</firstterm>,
+   which is created and dropped along with the subscription. This significantly
+   improves post-mortem analysis and operational visibility of the replication
+   setup.
+  </para>
+
   <sect2 id="logical-replication-subscription-slot">
    <title>Logical Replication Slot Management</title>
 
@@ -2009,6 +2023,9 @@ Included in publications:
    operations will simply be skipped.
   </para>
 
+  <sect2 id="logical-replication-conflict-logging">
+  <title>Conflict logging</title>
+
   <para>
    Additional logging is triggered, and the conflict statistics are collected (displayed in the
    <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view)
@@ -2121,8 +2138,111 @@ Included in publications:
     log.
   </para>
 
+  </sect2>
+
+  <sect2 id="logical-replication-conflict-table-based-logging">
+  <title>Table-based logging</title>
+
+  <para>
+   If <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>
+   is set to <literal>table</literal> or <literal>all</literal> then a dedicated conflict
+   log table will be automatically created. This table is created in the 
+   <literal>pg_conflict</literal> namespace. The name of the conflict log table
+   is <literal>pg_conflict_log_&lt;subid&gt;</literal>. The predefined schema of this table is
+   detailed in
+   <xref linkend="logical-replication-conflict-log-schema"/>.
+  </para>
+
+  <table id="logical-replication-conflict-log-schema">
+   <title>Conflict Log Table Schema</title>
+   <tgroup cols="3">
+    <thead>
+     <row>
+      <entry>Column</entry>
+      <entry>Type</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+    <tbody>
+     <row>
+      <entry><literal>relid</literal></entry>
+      <entry><type>oid</type></entry>
+      <entry>The OID of the local table where the conflict occurred.</entry>
+     </row>
+     <row>
+      <entry><literal>schemaname</literal></entry>
+      <entry><type>text</type></entry>
+      <entry>The schema name of the conflicting table.</entry>
+     </row>
+     <row>
+      <entry><literal>relname</literal></entry>
+      <entry><type>text</type></entry>
+      <entry>The name of the conflicting table.</entry>
+     </row>
+     <row>
+      <entry><literal>conflict_type</literal></entry>
+      <entry><type>text</type></entry>
+      <entry>The type of conflict that occurred (e.g., <literal>insert_exists</literal>).</entry>
+     </row>
+     <row>
+      <entry><literal>remote_xid</literal></entry>
+      <entry><type>xid</type></entry>
+      <entry>The remote transaction ID that caused the conflict.</entry>
+     </row>
+     <row>
+      <entry><literal>remote_commit_lsn</literal></entry>
+      <entry><type>pg_lsn</type></entry>
+      <entry>The final LSN of the remote transaction.</entry>
+     </row>
+     <row>
+      <entry><literal>remote_commit_ts</literal></entry>
+      <entry><type>timestamptz</type></entry>
+      <entry>The remote commit timestamp of the remote transaction.</entry>
+     </row>
+     <row>
+      <entry><literal>remote_origin</literal></entry>
+      <entry><type>text</type></entry>
+      <entry>The origin of the remote transaction.</entry>
+     </row>
+     <row>
+      <entry><literal>replica_identity</literal></entry>
+      <entry><type>json</type></entry>
+      <entry>The JSON representation of the replica identity.</entry>
+     </row>
+     <row>
+      <entry><literal>remote_tuple</literal></entry>
+      <entry><type>json</type></entry>
+      <entry>The JSON representation of the incoming remote row that caused the conflict.</entry>
+     </row>
+     <row>
+      <entry><literal>local_conflicts</literal></entry>
+      <entry><type>json[]</type></entry>
+      <entry>
+       An array of JSON objects representing the local state for each conflict attempt.
+       Each object includes the local transaction ID (<literal>xid</literal>), commit
+       timestamp (<literal>commit_ts</literal>), origin (<literal>origin</literal>),
+       conflicting key data (<literal>key</literal>), and the full local row
+       image (<literal>tuple</literal>).
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+  <para>
+   The conflicting row data, including the incoming remote row (<literal>remote_tuple</literal>)
+   and the associated local conflict details (<literal>local_conflicts</literal>), is stored in
+   <type>JSON</type> formats for flexible querying and analysis.
+  </para>
+
+  </sect2>
+  <sect2 id="logical-replication-conflict-file-based-logging">
+  <title>File-based logging</title>
+
   <para>
-   The log format for logical replication conflicts is as follows:
+   If <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>
+   is set to <literal>log</literal> or <literal>all</literal> then conflicts are logged to the server log using
+   the following format:
 <synopsis>
 LOG:  conflict detected on relation "<replaceable>schemaname</replaceable>.<replaceable>tablename</replaceable>": conflict=<replaceable>conflict_type</replaceable>
 DETAIL:  <replaceable class="parameter">detailed_explanation</replaceable>[: <replaceable class="parameter">detail_values</replaceable> [, ... ]].
@@ -2235,6 +2355,10 @@ DETAIL:  <replaceable class="parameter">detailed_explanation</replaceable>[: <re
    </variablelist>
   </para>
 
+  </sect2>
+  <sect2 id="logical-replication-conflict-notes">
+  <title>Notes</title>
+
   <para>
    Logical replication operations are performed with the privileges of the role
    which owns the subscription.  Permissions failures on target tables will
@@ -2311,6 +2435,8 @@ CONTEXT:  processing remote data for replication origin "pg_16395" during "INSER
    linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ...
    SKIP</command></link>.
   </para>
+
+ </sect2>
  </sect1>
 
  <sect1 id="logical-replication-restrictions">
@@ -2415,6 +2541,13 @@ CONTEXT:  processing remote data for replication origin "pg_16395" during "INSER
      key or replica identity defined for it.
     </para>
    </listitem>
+
+   <listitem>
+    <para>
+     Conflict log tables (see <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link> parameter)
+     are never published, even when using <literal>FOR ALL TABLES</literal> in a publication.
+    </para>
+   </listitem>
   </itemizedlist>
  </sect1>
 
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index e4f0b6b16c7..07b7ede52ec 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -293,8 +293,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
       <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>,
       <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>,
       <link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>,
-      <link linkend="sql-createsubscription-params-with-max-retention-duration"><literal>max_retention_duration</literal></link>, and
-      <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>.
+      <link linkend="sql-createsubscription-params-with-max-retention-duration"><literal>max_retention_duration</literal></link>,
+      <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, and
+      <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>.
       Only a superuser can set <literal>password_required = false</literal>.
      </para>
 
@@ -352,6 +353,14 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
       <quote><literal>pg_conflict_detection</literal></quote>, created to retain
       dead tuples for conflict detection, will be dropped.
      </para>
+
+     <para>
+      When the <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>
+      parameter is set to <literal>table</literal> or <literal>all</literal>, the system
+      automatically creates the internal conflict log table. Conversely, if the destination is changed to
+      <literal>log</literal>, logging to the table stops and the internal
+      table is automatically dropped.
+     </para>
     </listitem>
    </varlistentry>
 
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 07d5b1bd77c..82c651090ae 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -261,6 +261,55 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
         </listitem>
        </varlistentry>
 
+       <varlistentry id="sql-createsubscription-params-with-conflict-log-destination">
+        <term><literal>conflict_log_destination</literal> (<type>enum</type>)</term>
+        <listitem>
+         <para>
+          Specifies the destination for recording logical replication conflicts.
+         </para>
+         <para>
+          The available destinations are:
+          <itemizedlist>
+           <listitem>
+            <para>
+             <literal>log</literal>: Conflict details are recorded in the server log.
+             This is the default behavior.
+             See <xref linkend="logical-replication-conflict-file-based-logging"/> for details.
+            </para>
+           </listitem>
+           <listitem>
+            <para>
+             <literal>table</literal>: The system automatically creates a structured table
+             named <literal>pg_conflict_log_&lt;subid&gt;</literal> in the
+             <literal>pg_conflict</literal> schema. This allows for easy querying and
+             analysis of conflicts.
+             See <xref linkend="logical-replication-conflict-table-based-logging"/> for details.
+            </para>
+            <caution>
+             <para>
+              The internal conflict log table is strictly tied to the lifecycle of the
+              subscription or the <literal>conflict_log_destination</literal> setting. If
+              the subscription is dropped, or if the destination is changed to
+              <literal>log</literal>, the table and all its recorded conflict data are
+              <emphasis>permanently deleted</emphasis>.
+             </para>
+             <para>
+              If conflict history may be needed later, back up the conflict log table before
+              it gets removed.
+             </para>
+            </caution>
+           </listitem>
+           <listitem>
+            <para>
+             <literal>all</literal>: Records conflict details to both destinations
+             <literal>log</literal> and <literal>table</literal>.
+            </para>
+           </listitem>
+          </itemizedlist>
+         </para>
+        </listitem>
+       </varlistentry>
+
        <varlistentry id="sql-createsubscription-params-with-copy-data">
         <term><literal>copy_data</literal> (<type>boolean</type>)</term>
         <listitem>