Horton
Menu

Re: Perform streaming logical transactions by background workers and parallel apply

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, "shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2022-05-11T04:54:26Z
Lists: pgsql-hackers 
On Wed, May 11, 2022 at 9:17 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, May 10, 2022 at 6:10 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, May 10, 2022 at 10:35 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Wed, May 4, 2022 at 12:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > >
> > > > I think the other kind of problem that can happen here is delete
> > > > followed by an insert. If in the example provided by you, TX-1
> > > > performs delete (say it is large enough to cause streaming) and TX-2
> > > > performs insert then I think it will block the apply worker because
> > > > insert will start waiting infinitely. Currently, I think it will lead
> > > > to conflict due to insert but that is still solvable by allowing users
> > > > to remove conflicting rows.
> > > >
> > > > It seems both these problems are due to the reason that the table on
> > > > publisher and subscriber has different constraints otherwise, we would
> > > > have seen the same behavior on the publisher as well.
> > > >
> > > > There could be a few ways to avoid these and similar problems:
> > > > a. detect the difference in constraints between publisher and
> > > > subscribers like primary key and probably others (like whether there
> > > > is any volatile function present in index expression) when applying
> > > > the change and then we give ERROR to the user that she must change the
> > > > streaming mode to 'spill' instead of 'apply' (aka parallel apply).
> > > > b. Same as (a) but instead of ERROR just LOG this information and
> > > > change the mode to spill for the transactions that operate on that
> > > > particular relation.
> > >
> > > Given that it doesn't introduce a new kind of problem I don't think we
> > > need special treatment for that at least in this feature.
> > >
> >
> > Isn't the problem related to infinite wait by insert as explained in
> > my previous email (in the above-quoted text) a new kind of problem
> > that won't exist in the current implementation?
> >
>
> Sorry I had completely missed the point that the commit order won't be
> changed. I agree that this new implementation would introduce a new
> kind of issue as you mentioned above, and the opposite is not true.
>
> Regarding the case you explained in the previous email I also think it
> will happen with the parallel apply feature. The apply worker will be
> blocked until the conflict is resolved. I'm not sure how to avoid
> that. It would be not easy to compare constraints between publisher
> and subscribers when replicating partitioning tables.
>

I agree that partitioned tables need some more thought but in some
simple cases where replication happens via individual partition tables
(default), we can detect as we do for normal tables. OTOH, when
replication happens via root (publish_via_partition_root) it could be
tricky as the partitions could be different on both sides. I think the
cases where we can't safely identify the constraint difference won't
be considered for apply via a new bg worker.

-- 
With Regards,
Amit Kapila.

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix invalid memory access during the shutdown of the parallel apply worker.

  2. Fix assertion failure in apply worker.

  3. Use elog to report unexpected action in handle_streamed_transaction().

  4. Use appropriate wait event when sending data in the apply worker.

  5. Allow the logical_replication_mode to be used on the subscriber.

  6. Rename GUC logical_decoding_mode to logical_replication_mode.

  7. Display the leader apply worker's PID for parallel apply workers.

  8. Improve the code to decide and process the apply action.

  9. Document the newly added wait events added by commit 216a784829.

  10. Perform apply of large transactions by parallel workers.

  11. Wake up a subscription's replication worker processes after DDL.

  12. Add copyright notices to meson files

  13. Better document logical replication parameters

  14. Add a common function to generate the origin name.

  15. Harmonize parameter names in storage and AM code.

  16. Avoid using list_length() to test for empty list.

  17. Improve two comments related to a boolean DefElem's value

  18. Fix partition table's REPLICA IDENTITY checking on the subscriber.

  19. Fix data inconsistency between publisher and subscriber.

  20. Fix cache look-up failures while applying changes in logical replication.