Horton

Re: [HACKERS] logical decoding of two-phase transactions

Amit Kapila <amit.kapila16@gmail.com>

View thread View on postgr.es
From: Amit Kapila <amit.kapila16@gmail.com>
To: Ajin Cherian <itsajin@gmail.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>, Peter Smith <smithpb2250@gmail.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2020-12-19T03:14:51Z
Lists: pgsql-hackers

Attachments

On Fri, Dec 18, 2020 at 11:23 AM Ajin Cherian <itsajin@gmail.com> wrote:
>
> On Thu, Dec 17, 2020 at 11:47 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > I went ahead and used both origin_lsn and origin_timestamp to avoid
> > the possibility of a match of prepared xact from two different nodes.
> > We can handle this at begin_prepare and prepare time but we don't have
> > prepare_lsn and prepare_timestamp at rollback_prepared time, so what
> > do about that? As of now, I am using just GID at rollback_prepare time
> > and that would have been sufficient if we always receive prepare
> > before rollback because at prepare time we would have checked
> > origin_lsn and origin_timestamp. But it is possible that we get
> > rollback prepared without prepare in case if prepare happened before
> > consistent_snapshot is reached and rollback happens after that. For
> > commit-case, we do send prepare and all the data at commit time in
> > such a case but doing so for rollback case doesn't sound to be a good
> > idea. Another possibility is that we send prepare_lsn and prepare_time
> > in rollback_prepared API to deal with this. I am not sure if it is a
> > good idea to just rely on GID in rollback_prepare. What do you think?
>
> Thinking about it for some time, my initial reaction was that the
> distributed servers should maintain uniqueness of GIDs and re-checking
> with LSNs is just overkill. But thinking some more, I realise that
> since we allow reuse of GIDs, there could be a race condition where a
> previously aborted/committed txn's GID was reused
> which could lead to this. Yes, I think we could change
> rollback_prepare to send out prepare_lsn and prepare_time as well,
> just to be safe.
>

Okay, I have changed the rollback_prepare API as discussed above and
accordingly handle the case where rollback is received without prepare
in apply_handle_rollback_prepared.

While testing for this case, I noticed that the tracking of
replication progress for aborts is not complete due to which after
restart we can again ask for the rollback lsn. This shouldn't be a
problem with the latest code because we will simply skip it when there
is no corresponding prepare but this is far from ideal because that is
the sole purpose of tracking via replication origins. This was due to
the incomplete handling of aborts in the original commit 1eb6d6527a. I
have fixed this now in a separate patch
v33-0004-Track-replication-origin-progress-for-rollbacks. If you want
to see the problem then change the below code and don't apply
v33-0004-Track-replication-origin-progress-for-rollbacks, the
regression failure is due to the reason that we are not tracking
progress for aborts:

apply_handle_rollback_prepared
{
..
if (LookupGXact(rollback_data.gid, rollback_data.prepare_end_lsn,
rollback_data.preparetime))
..
}

to
apply_handle_rollback_prepared
{
..
Assert (LookupGXact(rollback_data.gid, rollback_data.prepare_end_lsn,
rollback_data.preparetime));

-- 
With Regards,
Amit Kapila.

Commits

  1. Add prepare API support for streaming transactions in logical replication.

  2. Doc: minor improvements for logical replication protocol documentation.

  3. Fix test failure in 021_twophase.pl.

  4. Refactor to make common functions in proto.c and worker.c.

  5. Unify PostgresNode's new() and get_new_node() methods

  6. Fix potential buffer overruns in proto.c.

  7. Add support for prepared transactions to built-in logical replication.

  8. Refactor function parse_subscription_options.

  9. Allow enabling two-phase option via replication protocol.

  10. Don't use Asserts to check for violations of replication protocol.

  11. Improve psql tab completion for options of subcriptions and publications

  12. Rearrange logrep worker's snapshot handling some more.

  13. doc: Update information of new messages for logical replication.

  14. ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION

  15. Allow pgoutput to send logical decoding messages.

  16. Refactor function parse_output_parameters.

  17. Avoid repeated decoding of prepared transactions after a restart.

  18. Fix an oversight in ReorderBufferFinishPrepared.

  19. Allow multiple xacts during table sync in logical replication.

  20. Fix replication of in-progress transactions in tablesync worker.

  21. Fix 'skip-empty-xacts' option in test_decoding for streaming mode.

  22. Use Enum for top level logical replication message types.

  23. Add support for streaming to built-in logical replication.

  24. Fix the logical streaming test.

  25. Implement streaming mode in ReorderBuffer.

  26. Extend the logical decoding output plugin API with stream methods.

  27. Store 2PC GID in commit/abort WAL recs for logical decoding