Thread

1. Local visibility with logical decoding

   Marko Tiikkaja <marko@joh.to> — 2020-07-20T14:27:30Z

   Hi,
   
   It appears that when logical decoding sends out the data from the output
   plugin, it is not guaranteed that the decoded transaction's effects are
   visible on the source server.  Is this the way it's supposed to work?
   
   If so, would doing something like this in the output plugin be reasonable?
   
       TransactionId xid = transaction->xid;
       if (transaction->is_known_as_subxact)
           xid = transaction->toplevel_xid;
   
       if (TransactionIdIsInProgress(xid))
           XactLockTableWait(xid, NULL, NULL, XLTW_None);
   
   
   -marko
   

2. Re: Local visibility with logical decoding

   Antonin Houska <ah@cybertec.at> — 2020-07-20T16:38:00Z

   Marko Tiikkaja <marko@joh.to> wrote:
   
   > It appears that when logical decoding sends out the data from the output
   > plugin, it is not guaranteed that the decoded transaction's effects are
   > visible on the source server.  Is this the way it's supposed to work?
   
   Can you please share the test that indicates this behavior? As far as I
   understand, the transaction must have been committed before the output plugin
   starts to receive the changes.
   
   -- 
   Antonin Houska
   Web: https://www.cybertec-postgresql.com
   
   
   
   

3. Re: Local visibility with logical decoding

   Marko Tiikkaja <marko@joh.to> — 2020-07-20T17:10:55Z

   On Mon, Jul 20, 2020 at 7:36 PM Antonin Houska <ah@cybertec.at> wrote:
   
   > Marko Tiikkaja <marko@joh.to> wrote:
   >
   > > It appears that when logical decoding sends out the data from the output
   > > plugin, it is not guaranteed that the decoded transaction's effects are
   > > visible on the source server.  Is this the way it's supposed to work?
   >
   > Can you please share the test that indicates this behavior? As far as I
   > understand, the transaction must have been committed before the output
   > plugin
   > starts to receive the changes.
   >
   
   I don't have a reliable test program, but you can reproduce quite easily
   with test_decoding if you put a breakpoint before the SyncRepWaitForLSN()
   call in src/backend/access/transam/xact.c.  pg_logicalrecv will see the
   changes while the session is sitting on the breakpoint, and not finishing
   its commit.
   
   
   -marko
   

4. Re: Local visibility with logical decoding

   Andres Freund <andres@anarazel.de> — 2020-07-20T20:21:34Z

   Hi,
   
   On 2020-07-20 17:27:30 +0300, Marko Tiikkaja wrote:
   > It appears that when logical decoding sends out the data from the output
   > plugin, it is not guaranteed that the decoded transaction's effects are
   > visible on the source server.  Is this the way it's supposed to work?
   
   At the moment the visibility behaviour is basically the same as crash
   recovery / standbys. And they just look at the WAL...
   
   
   > If so, would doing something like this in the output plugin be reasonable?
   > 
   >     TransactionId xid = transaction->xid;
   >     if (transaction->is_known_as_subxact)
   >         xid = transaction->toplevel_xid;
   > 
   >     if (TransactionIdIsInProgress(xid))
   >         XactLockTableWait(xid, NULL, NULL, XLTW_None);
   
   I'd not be surprised if this had a potential to cause deadlocks.
   
   Greetings,
   
   Andres Freund