Horton

Re: logical decoding and replication of sequences, take 2

Dilip Kumar <dilipbalaut@gmail.com>

View thread
From: Dilip Kumar <dilipbalaut@gmail.com>
To: Tomas Vondra <tomas.vondra@enterprisedb.com>
Cc: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Amit Kapila <amit.kapila16@gmail.com>, "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>, Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Masahiko Sawada <sawada.mshk@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Date: 2023-12-07T05:11:01Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Migrate logical slots to the new node during an upgrade.

  2. Make test_decoding ddl.out shorter

  3. Fix snapshot handling in logicalmsg_decode

  4. doc: Adjust a few more references to "postmaster"

  5. Revert "Logical decoding of sequences"

On Wed, Dec 6, 2023 at 7:09 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> Yes, if something like this happens, that'd be a problem:
>
> 1) decoding starts, with
>
>    SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT
>
> 2) transaction that creates a new refilenode gets decoded, but we skip
>    it because we don't have the correct snapshot
>
> 3) snapshot changes to SNAPBUILD_FULL_SNAPSHOT
>
> 4) we decode sequence change from nextval() for the sequence
>
> This would lead to us attempting to apply sequence change for a
> relfilenode that's not visible yet (and may even get aborted).
>
> But can this even happen? Can we start decoding in the middle of a
> transaction? How come this wouldn't affect e.g. XLOG_HEAP2_NEW_CID,
> which is also skipped until SNAPBUILD_FULL_SNAPSHOT. Or logical
> messages, where we also call the output plugin in non-transactional cases.

It's not a problem for logical messages because whether the message is
transaction or non-transactional is decided while WAL logs the message
itself.  But here our problem starts with deciding whether the change
is transactional vs non-transactional, because if we insert the
'relfilenode' in hash then the subsequent sequence change in the same
transaction would be considered transactional otherwise
non-transactional.  And XLOG_HEAP2_NEW_CID is just for changing the
snapshot->curcid which will only affect the catalog visibility of the
upcoming operation in the same transaction, but that's not an issue
because if some of the changes of this transaction are seen when
snapbuild state < SNAPBUILD_FULL_SNAPSHOT then this transaction has to
get committed before the state change to SNAPBUILD_CONSISTENT_SNAPSHOT
i.e. the commit LSN of this transaction is going to be <
start_decoding_at.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com