RE: Perform streaming logical transactions by background workers and parallel apply

Wei Wang (Fujitsu) <wangw.fnst@fujitsu.com>

From: "wangw.fnst@fujitsu.com" <wangw.fnst@fujitsu.com>

To: Amit Kapila <amit.kapila16@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>

Cc: "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, Peter Smith <smithpb2250@gmail.com>, shveta malik <shveta.malik@gmail.com>, "shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Dilip Kumar <dilipbalaut@gmail.com>

Date: 2023-01-18T04:48:01Z

Lists: pgsql-hackers

Attachments

v84-0001-Stop-extra-worker-if-GUC-was-changed.patch (application/octet-stream) patch v84-0001
v84-0002-Add-GUC-stream_serialize_threshold-and-test-seri.patch (application/octet-stream) patch v84-0002
v84-0003-Retry-to-apply-streaming-xact-only-in-apply-work.patch (application/octet-stream) patch v84-0003

On Wed, Jan 18, 2023 12:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Tue, Jan 17, 2023 at 8:07 PM Masahiko Sawada <sawada.mshk@gmail.com>
> wrote:
> >
> > On Tue, Jan 17, 2023 at 6:14 PM houzj.fnst@fujitsu.com
> > <houzj.fnst@fujitsu.com> wrote:
> > >
> > > On Tuesday, January 17, 2023 2:46 PM Masahiko Sawada
> <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Tue, Jan 17, 2023 at 12:37 PM houzj.fnst@fujitsu.com
> > > > <houzj.fnst@fujitsu.com> wrote:
> > > > I'm slightly concerned that there could be overhead of executing
> > > > GetLeaderApplyWorkerPid () for every backend process except for parallel
> > > > query workers. The number of such backends could be large and
> > > > GetLeaderApplyWorkerPid() acquires the lwlock. For example, does it
> make
> > > > sense to check (st_backendType == B_BG_WORKER) before calling
> > > > GetLeaderApplyWorkerPid()? Or it might not be a problem since it's
> > > > LogicalRepWorkerLock which is not likely to be contended.
> > >
> > > Thanks for the comment and I think your suggestion makes sense.
> > > I have added the check before getting the leader pid. Here is the new
> version patch.
> >
> > Thank you for updating the patch. Looks good to me.
> >
> 
> Pushed.

Rebased and attach remaining patches for reviewing.

Regards,
Wang Wei

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Fix invalid memory access during the shutdown of the parallel apply worker.
- 3d144c6c8602 16.0 landed
Fix assertion failure in apply worker.
- de63f8dadee4 16.0 landed
Use elog to report unexpected action in handle_streamed_transaction().
- 781ac42d43ab 16.0 landed
Use appropriate wait event when sending data in the apply worker.
- d9d7fe68d35e 16.0 landed
Allow the logical_replication_mode to be used on the subscriber.
- 9f2213a7c575 16.0 landed
Rename GUC logical_decoding_mode to logical_replication_mode.
- 1e8b61735cfb 16.0 landed
Display the leader apply worker's PID for parallel apply workers.
- d540a02a724b 16.0 landed
Improve the code to decide and process the apply action.
- c981d9145dea 16.0 landed
Document the newly added wait events added by commit 216a784829.
- cd06ccd78fcf 16.0 landed
Perform apply of large transactions by parallel workers.
- 216a784829c2 16.0 landed
Wake up a subscription's replication worker processes after DDL.
- c6e1f62e2cee 16.0 cited
Add copyright notices to meson files
- 8284cf5f746f 16.0 cited
Better document logical replication parameters
- a8500750ca0a 16.0 cited
Add a common function to generate the origin name.
- 776e1c8a5d14 16.0 cited
Harmonize parameter names in storage and AM code.
- bfcf1b34805f 16.0 cited
Avoid using list_length() to test for empty list.
- efd0c16becbf 16.0 cited
Improve two comments related to a boolean DefElem's value
- 8445f5a21d40 16.0 cited
Fix partition table's REPLICA IDENTITY checking on the subscriber.
- 26b3455afaa9 15.0 cited
Fix data inconsistency between publisher and subscriber.
- b7658c24c7c1 15.0 cited
Fix cache look-up failures while applying changes in logical replication.
- 5a97b1325453 15.0 cited