Horton

Thread

  1. Fix logical decoding not track transaction during SNAPBUILD_BUILDING_SNAPSHOT

    ocean_li_996 <ocean_li_996@163.com> — 2025-11-22T09:28:13Z

    Hi all,


I would like to share a logical replication bug and some possible fixes. It seems that this bug has existed since
logical replication was first introduced, so it has been around for quite some time. In fact, the previously
reported issues [1], [2], [3] were all caused by this bug.


# Problem description


When in the BUILDING_SNAPSHOT state, the snapshot builder does not track the status of any
transaction. It can lead to missing transaction states when:
-- The transaction commits before the builder reaches FULL_SNAPSHOT state, and
-- The transaction's xid is greater than or equal to builder->xmin when the builder reaches
FULL_SNAPSHOT state.


Once in FULL_SNAPSHOT state, the builder constructs a base snapshot using incomplete transaction state
information. This results in an incorrect base snapshot, which can cause unpredictable behavior during
subsequent decoding. The case provided in v6-0002 attachment reproduces the issue (provided by ChangAo Chen).


# Code-level analysis


SnapBuildCommitTxn does consider transaction processing during the BUILDING_SNAPSHOT state. However, it
is only called from xact_decode -> DecodeCommit. xact_decode does not process any xact record when snapshot
builder have not yet reached the FULL_SNAPSHOT state, meaning those commits are ignored. Similarly, other
functions marking transaction having catalog changes (e.g., heap2_decode) also do not handle records before
reaching the FULL_SNAPSHOT state.


# Possible fixes


1. Replace snapshot at the time we reach CONSISTENT state.


Ajin Cherian in [4] and my initial thought was that although the snapshot at FULL_SNAPSHOT state might be
wrong, the snapshot at CONSISTENT state is guaranteed to be correct. Since decoding always starts after
reaching CONSISTENT state, we could update both the reorder buffer and the builder snapshot with the one
captured at CONSISTENT state. However, IMUC, this would still cause changes generated before CONSISTENT to
carry a wrong snapshot (see SnapBuildDistributeSnapshotAndInval).


2. Track transactions during BUILDING_SNAPSHOT state for snapshot builder
If the builder does not track transactions in BUILDING_SNAPSHOT state, then we make it track them.


1) ChangAo Chen in v6-0001 attachment provided a fix, already reviewed by several people (including me).
Bertrand Drouvot in [5] considered the logic a bit messy. And I prefer we should make the behavior of
snapshot building similar in both BUILDING_SNAPSHOT and FULL_SNAPSHOT states, except in cases where
a base snapshot is needed.


2) Based on v6-0001, I have provided a minimal fix in v6-0003 (not yet reviewed). AFAICS, it resolves
the problem, though it records additional useless information in the reorder buffer during BUILDING_SNAPSHOT
state (which is discarded later). This increases memory usage and slightly impacts performance. But since
snapshot building is infrequent, I consider this acceptable.


3) I have also prepared a cleaner and more efficient fix in v6-0004 than v6-0003, albeit more complex
(similar to v6-0001). Provided as an alternative reference.


I think we should fix this issue to ensure snapshot building is correct.
Looking forward to your reviews and any feedback on the above proposed solutions.


Best regards,
Haiyang Li




[1] https://www.postgresql.org/message-id/tencent_6AAF072A7623A11A85C0B5FD290232467808%40qq.com
[2] https://www.postgresql.org/message-id/18509-983f064d174ea880@postgresql.org
[3] https://www.postgresql.org/message-id/2b9e5ac8.136f.19a8f7297ee.Coremail.ocean_li_996%40163.com
[4] https://www.postgresql.org/message-id/CAFPTHDYSQipcO_%2BGNt-ZQsk6cidt9Lc4PkcdvO7jnrugiUw0eg%40mail.gmail.com 
[5] https://www.postgresql.org/message-id/ZrnlgJEH473Q1kTp%40ip-10-97-1-34.eu-west-3.compute.internal

  2. Re:Fix logical decoding not track transaction during SNAPBUILD_BUILDING_SNAPSHOT

    ocean_li_996 <ocean_li_996@163.com> — 2025-11-22T09:33:59Z

    Hi Ajin & Bertrand,



I missed CC you in last email. Just for your infomation. No defence.


Best regards,
Haiyang Li

  3. Re: Fix logical decoding not track transaction duringSNAPBUILD_BUILDING_SNAPSHOT

    cca5507 <cca5507@qq.com> — 2025-11-22T10:34:23Z

    Hi Haiyang,

Thank you for your summary.

One important thing is that we must not skip any call to ReorderBufferXidSetCatalogChanges() (direct or indirect) during fast-forwarding or building snapshot, because the historic snapshot only tracks txns with catalog changes, the v6-0004 seems to skip it in xact_decode().

Here is a related bug:

https://www.postgresql.org/message-id/flat/tencent_3A071B760AA1A38540B57F297332B7781C08%40qq.com

--
Regards,
ChangAo Chen

  4. Re: Fix logical decoding not track transaction duringSNAPBUILD_BUILDING_SNAPSHOT

    ocean_li_996 <ocean_li_996@163.com> — 2025-11-22T16:05:49Z

    Hi ChaoAo,
At 2025-11-22 18:34:23, "cca5507" <cca5507@qq.com> wrote:
>One important thing is that we must not skip any call to ReorderBufferXidSetCatalogChanges() (direct or indirect) during fast-forwarding or building snapshot, because the historic snapshot only tracks txns with catalog changes, the v6-0004 seems to skip it in xact_decode().

>
v6-0004 only skip the transaction commited during START state and precedeing next_phase_at
(set when changing to BUILDING_SNAPSHOT state) during BUILDING_SNAPSHOT. Those transactions
are always useless no matter in fast forward or not. Plaese recheck v6-0004 again.


>Here is a related bug: > >https://www.postgresql.org/message-id/flat/tencent_3A071B760AA1A38540B57F297332B7781C08%40qq.com >


Yeah, I have researched that issue. I think your analyze is correct for me. But it is
independent of this thread. Let's discuss it in thread where the issues belongs to.


Best regards,
Haiyang Li

  5. Re:Fix logical decoding not track transaction during SNAPBUILD_BUILDING_SNAPSHOT

    ocean_li_996 <ocean_li_996@163.com> — 2025-11-23T06:05:05Z

    Hi,


Sorry for the direct CC. 


Given your expertise in logical replication and your dedication to improving its functionality, 
I think the issue mentioned in [1] may be worth some attention. If you have time, I am appreciate 
your thoughts or opinions on the issue.


Thanks and sorry again for the intrusion.


Best regards,
Haiyang Li


[1] https://www.postgresql.org/message-id/3575444b.25e0.19aaae481e0.Coremail.ocean_li_996%40163.com

  6. Re: Fix logical decoding not track transaction during SNAPBUILD_BUILDING_SNAPSHOT

    cca5507 <cca5507@qq.com> — 2025-11-23T08:15:35Z

    Hi Haiyang,

> v6-0004 only skip the transaction commited during START state and precedeing next_phase_at
> (set when changing to BUILDING_SNAPSHOT state) during BUILDING_SNAPSHOT. Those transactions
> are always useless no matter in fast forward or not. Plaese recheck v6-0004 again.

Yeah, you're right. What's useful for building the snapshot and we currently don't track are the txns
start after BUILDING_SNAPSHOT and commit before FULL_SNAPSHOT, their xids all >= next_phase_at
during BUILDING_SNAPSHOT.

--
Regards,
ChangAo Chen