Horton
Menu

RE: Perform streaming logical transactions by background workers and parallel apply

Wei Wang (Fujitsu) <wangw.fnst@fujitsu.com>

View thread
From: "wangw.fnst@fujitsu.com" <wangw.fnst@fujitsu.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, Peter Smith <smithpb2250@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, "shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2022-12-28T04:38:58Z
Lists: pgsql-hackers

Attachments

On Tue, Dec 27, 2022 19:37 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Tue, Dec 27, 2022 at 10:24 AM wangw.fnst@fujitsu.com
> <wangw.fnst@fujitsu.com> wrote:
> >
> > Attach the new version patch which addressed all above comments and part
> of
> > comments from [1] except one comment that are being discussed.
> >

Thanks for your comments.

> 1.
> +# Test that the deadlock is detected among leader and parallel apply workers.
> +
> +$node_subscriber->append_conf('postgresql.conf', "deadlock_timeout =
> 1ms");
> +$node_subscriber->reload;
> +
> 
> A. I see that the other existing tests have deadlock_timeout set as
> 10ms, 100ms, 100s, etc. Is there a reason to keep so low here? Shall
> we keep it as 10ms?

No, I think you are right. Keep it as 10ms.

> B. /among leader/among the leader

Fixed.

> 2. Can we leave having tests in 022_twophase_cascade to be covered by
> parallel mode? The two-phase and parallel apply will be covered by
> 023_twophase_stream, so not sure if we get any extra coverage by
> 022_twophase_cascade.

Compared with 023_twophase_stream, there is "rollback a subtransaction" in
022_twophase_cascade, but since this part of the code can be covered by tests
in 018_stream_subxact_abort, I think we can remove parallel version for
022_twophase_cascade. So I reverted changes in 022_twophase_cascade for
parallel mode and added some comments atop this file.

> 3. Let's combine 0001 and 0002 as both have got reviewed independently.

Combined them into one patch.

And I also checked and merged the diff patch in [1].

Besides, also fixed the below problem:
In previous versions, we didn't wait for STREAM_ABORT transactions to complete.
But in extreme cases, this can cause problems if the STREAM_ABORT transaction
doesn't complete and xid wraparound occurs on the publisher-side. Fixed this by
waiting for the STREAM_ABORT transaction to complete.

Attach the new patch set.

[1] - https://www.postgresql.org/message-id/CAA4eK1%2B5gTjHzWovkbUj%2BxsQ9yO9jVcKsS-3c5ZXLFy8JmfT%3DA%40mail.gmail.com

Regards,
Wang wei

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix invalid memory access during the shutdown of the parallel apply worker.

  2. Fix assertion failure in apply worker.

  3. Use elog to report unexpected action in handle_streamed_transaction().

  4. Use appropriate wait event when sending data in the apply worker.

  5. Allow the logical_replication_mode to be used on the subscriber.

  6. Rename GUC logical_decoding_mode to logical_replication_mode.

  7. Display the leader apply worker's PID for parallel apply workers.

  8. Improve the code to decide and process the apply action.

  9. Document the newly added wait events added by commit 216a784829.

  10. Perform apply of large transactions by parallel workers.

  11. Wake up a subscription's replication worker processes after DDL.

  12. Add copyright notices to meson files

  13. Better document logical replication parameters

  14. Add a common function to generate the origin name.

  15. Harmonize parameter names in storage and AM code.

  16. Avoid using list_length() to test for empty list.

  17. Improve two comments related to a boolean DefElem's value

  18. Fix partition table's REPLICA IDENTITY checking on the subscriber.

  19. Fix data inconsistency between publisher and subscriber.

  20. Fix cache look-up failures while applying changes in logical replication.