Horton

Re: long-standing data loss bug in initial sync of logical replication

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>, Shlok Kyal <shlok.kyal.oss@gmail.com>, vignesh C <vignesh21@gmail.com>, Nitin Motiani <nitinmotiani@google.com>, Andres Freund <andres@anarazel.de>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2025-02-28T06:42:12Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix typo in test file name added in commit 4909b38af0.

  2. Fix data loss in logical replication.

  3. Avoid invalidating all RelationSyncCache entries on publication rename.

  4. Remove obsolete RECHECK keyword completely

  5. Backport BackgroundPsql perl test module

Attachments

On Fri, Feb 28, 2025 at 9:45 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Feb 28, 2025 at 6:15 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Thu, Feb 27, 2025 at 12:14 AM Zhijie Hou (Fujitsu)
> > >
> > > I think distributing invalidations to a transaction that has not yet built a
> > > base snapshot is un-necessary. This is because, during the process of building
> > > its base snapshot, such a transaction will have already recorded the XID of the
> > > transaction that altered the publication information into its array of
> > > committed XIDs. Consequently, it will reflect the latest changes in the catalog
> > > from the beginning. In the context of logical decoding, this scenario is
> > > analogous to decoding a new transaction initiated after the catalog-change
> > > transaction has been committed.
> > >
> > > The original issue arises because the catalog cache was constructed using an
> > > outdated snapshot that could not reflect the latest catalog changes. However,
> > > this is not a problem in cases without a base snapshot. Since the existing
> > > catalog cache should have been invalidated upon decoding the committed
> > > catalog-change transaction, the subsequent transactions will construct a new
> > > cache with the latest snapshot.
> >
> > I've also concluded it's not necessary but the reason and analysis
> > might be somewhat different.
>

Based on the discussion on this point and Hou-San's proposed comment,
I have tried to add/edit a few comments in 0001 patch. See, if those
make sense to you, it is important to capture the reason and theory we
discussed here in the form of comments so that it will be easy to
remember the reason in the future.

-- 
With Regards,
Amit Kapila.