Horton

Re: Asynchronous Append on postgres_fdw nodes.

Craig Ringer <craig.ringer@enterprisedb.com>

View thread View on postgr.es
From: Craig Ringer <craig.ringer@enterprisedb.com>
To: "movead.li@highgo.ca" <movead.li@highgo.ca>
Cc: Etsuro Fujita <etsuro.fujita@gmail.com>, Horiguchi <horikyota.ntt@gmail.com>, Justin Pryzby <pryzby@telsasoft.com>, "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>, Ahsan Hadi <ahsan.hadi@gmail.com>, pgsql-hackers <pgsql-hackers@lists.postgresql.org>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, tsunakawa.takay@fujitsu.com
Date: 2020-11-30T02:45:34Z
Lists: pgsql-hackers 
"On Thu, Nov 26, 2020 at 9:28 AM movead.li@highgo.ca
<movead.li@highgo.ca> wrote:
>
>
> I test the patch and occur several issues as blow:
>
> Issue one:
> Get a Assert error at 'Assert(bms_is_member(i, node->as_needrequest));' in
> ExecAppendAsyncRequest() function when I use more than two foreign table
> on different foreign server.
>
> I research the code and do such change then the Assert problem disappear.
>
> @@ -1004,6 +1004,7 @@ ExecAppendAsyncResponse(AsyncRequest *areq) bms_del_member(node->as_needrequest, areq->request_index); node->as_asyncpending = bms_add_member(node->as_asyncpending, areq->request_index); + node->as_lastasyncplan = INVALID_SUBPLAN_INDEX; return false; }
>
> Issue two:
> Then I test and find if I have sync subplan and async sunbplan, it will run over
> the sync subplan then the async turn, I do not know if it is intent.

I only just noticed this patch. It's very interesting to me given the
ongoing work happening on postgres_fdw batching and the way libpq
pipelining is looking like it's getting there. I'll study up on the
executor and see if I can understand this well enough to hack together
a PoC to make it use libpq batching.

Have you taken a look at how this patch may overlap with those?

See -hackers threads:

* "POC: postgres_fdw insert batching" [1]
* "PATCH: Batch/pipelining support for libpq" [2]

[1] https://www.postgresql.org/message-id/OSBPR01MB2982039EA967F0304CC6A3ECFE0B0@OSBPR01MB2982.jpnprd01.prod.outlook.com
[2] https://www.postgresql.org/message-id/20201026190936.GA18705@alvherre.pgsql

Commits

  1. Improve comments for trivial_subqueryscan().

  2. Disable asynchronous execution if using gating Result nodes.

  3. Allow asynchronous execution in more cases.

  4. Doc: Further update documentation for asynchronous execution.

  5. Fix rescanning of async-aware Append nodes.

  6. Doc: Update documentation for asynchronous execution.

  7. Fix EXPLAIN ANALYZE for async-capable nodes.

  8. Minor code cleanup in asynchronous execution support.

  9. Adjust input value to WaitEventSetWait() in ExecAppendAsyncEventWait().

  10. Add support for asynchronous execution.