Thread

Commits

Prevent asynchronous execution of direct foreign-table modifications.
- a784859f4480 14.0 landed

Inherited UPDATE/DELETE vs async execution

Etsuro Fujita <etsuro.fujita@gmail.com> — 2021-05-07T16:20:51Z

I noticed this while working on the
EXPLAIN-ANALYZE-for-async-capable-nodes issue:

EXPLAIN (VERBOSE, COSTS OFF)
DELETE FROM async_pt;
                           QUERY PLAN
----------------------------------------------------------------
 Delete on public.async_pt
   Foreign Delete on public.async_p1 async_pt_1
   Foreign Delete on public.async_p2 async_pt_2
   Delete on public.async_p3 async_pt_3
   ->  Append
         ->  Async Foreign Delete on public.async_p1 async_pt_1
               Remote SQL: DELETE FROM public.base_tbl1
         ->  Async Foreign Delete on public.async_p2 async_pt_2
               Remote SQL: DELETE FROM public.base_tbl2
         ->  Seq Scan on public.async_p3 async_pt_3
               Output: async_pt_3.tableoid, async_pt_3.ctid
(11 rows)

DELETE FROM async_pt;
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
connection to server was lost

The cause for this would be that direct-update plans are mistakenly
treated as async-capable ones, as shown in the EXPLAIN output.  To
fix, I think we should modify postgresPlanDirectModify() so that it
clears the async-capable flag if it is set.  Attached is a patch for
that.  Maybe I am missing something, though.

Best regards,
Etsuro Fujita

Re: Inherited UPDATE/DELETE vs async execution

Justin Pryzby <pryzby@telsasoft.com> — 2021-05-10T00:20:44Z

On Sat, May 08, 2021 at 01:20:51AM +0900, Etsuro Fujita wrote:
> I noticed this while working on the
> EXPLAIN-ANALYZE-for-async-capable-nodes issue:
> 
> DELETE FROM async_pt;
> server closed the connection unexpectedly

Confirmed, +Tomas, and added at
https://wiki.postgresql.org/wiki/PostgreSQL_14_Open_Items

-- 
Justin

Re: Inherited UPDATE/DELETE vs async execution

Etsuro Fujita <etsuro.fujita@gmail.com> — 2021-05-10T02:20:09Z

On Mon, May 10, 2021 at 9:20 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> On Sat, May 08, 2021 at 01:20:51AM +0900, Etsuro Fujita wrote:
> > I noticed this while working on the
> > EXPLAIN-ANALYZE-for-async-capable-nodes issue:
> >
> > DELETE FROM async_pt;
> > server closed the connection unexpectedly
>
> Confirmed, +Tomas, and added at
> https://wiki.postgresql.org/wiki/PostgreSQL_14_Open_Items

Thanks for that!

Maybe my explanation was not good, but actually, this is a follow-up
for commits 27e1f1456 and 86dc90056, which were independently
discussed and committed, and IIUC, the batch-insert work by Tomas
would not be related to this.  So I’ll work on it unless Tom (or
anyone else) wants to.

Best regards,
Etsuro Fujita

Re: Inherited UPDATE/DELETE vs async execution

Amit Langote <amitlangote09@gmail.com> — 2021-05-10T12:20:52Z

Fujita-san,

On Sat, May 8, 2021 at 1:21 AM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> I noticed this while working on the
> EXPLAIN-ANALYZE-for-async-capable-nodes issue:
>
> EXPLAIN (VERBOSE, COSTS OFF)
> DELETE FROM async_pt;
>                            QUERY PLAN
> ----------------------------------------------------------------
>  Delete on public.async_pt
>    Foreign Delete on public.async_p1 async_pt_1
>    Foreign Delete on public.async_p2 async_pt_2
>    Delete on public.async_p3 async_pt_3
>    ->  Append
>          ->  Async Foreign Delete on public.async_p1 async_pt_1
>                Remote SQL: DELETE FROM public.base_tbl1
>          ->  Async Foreign Delete on public.async_p2 async_pt_2
>                Remote SQL: DELETE FROM public.base_tbl2
>          ->  Seq Scan on public.async_p3 async_pt_3
>                Output: async_pt_3.tableoid, async_pt_3.ctid
> (11 rows)
>
> DELETE FROM async_pt;
> server closed the connection unexpectedly
>     This probably means the server terminated abnormally
>     before or while processing the request.
> connection to server was lost
>
> The cause for this would be that direct-update plans are mistakenly
> treated as async-capable ones, as shown in the EXPLAIN output.

I guess that happens because the ForeignScan nodes responsible for
scanning or direct-updating/deleting from child foreign tables appear
under an Append as of 86dc90056, whereas before they would appear as
child plans of a ModifyTable node.  IIUC, it's the Append that causes
the async_capable flag to be set in those ForeignScan nodes.

>  To
> fix, I think we should modify postgresPlanDirectModify() so that it
> clears the async-capable flag if it is set.  Attached is a patch for
> that.  Maybe I am missing something, though.

I see that your patch is to disable asynchronous execution in
ForeignScan nodes responsible for direct update/delete, but why not do
the same for other ForeignScan nodes too?  Or the other way around --
is it because fixing the crash that occurs in the former's case would
be a significant undertaking for little gain?

--
Amit Langote
EDB: http://www.enterprisedb.com

Re: Inherited UPDATE/DELETE vs async execution

Etsuro Fujita <etsuro.fujita@gmail.com> — 2021-05-11T08:56:46Z

Amit-san,

On Mon, May 10, 2021 at 9:21 PM Amit Langote <amitlangote09@gmail.com> wrote:
> On Sat, May 8, 2021 at 1:21 AM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> > I noticed this while working on the
> > EXPLAIN-ANALYZE-for-async-capable-nodes issue:
> >
> > EXPLAIN (VERBOSE, COSTS OFF)
> > DELETE FROM async_pt;
> >                            QUERY PLAN
> > ----------------------------------------------------------------
> >  Delete on public.async_pt
> >    Foreign Delete on public.async_p1 async_pt_1
> >    Foreign Delete on public.async_p2 async_pt_2
> >    Delete on public.async_p3 async_pt_3
> >    ->  Append
> >          ->  Async Foreign Delete on public.async_p1 async_pt_1
> >                Remote SQL: DELETE FROM public.base_tbl1
> >          ->  Async Foreign Delete on public.async_p2 async_pt_2
> >                Remote SQL: DELETE FROM public.base_tbl2
> >          ->  Seq Scan on public.async_p3 async_pt_3
> >                Output: async_pt_3.tableoid, async_pt_3.ctid
> > (11 rows)
> >
> > DELETE FROM async_pt;
> > server closed the connection unexpectedly
> >     This probably means the server terminated abnormally
> >     before or while processing the request.
> > connection to server was lost
> >
> > The cause for this would be that direct-update plans are mistakenly
> > treated as async-capable ones, as shown in the EXPLAIN output.
>
> I guess that happens because the ForeignScan nodes responsible for
> scanning or direct-updating/deleting from child foreign tables appear
> under an Append as of 86dc90056, whereas before they would appear as
> child plans of a ModifyTable node.  IIUC, it's the Append that causes
> the async_capable flag to be set in those ForeignScan nodes.

That's right.

The inherited update/delete work is great!  Thanks for that!

> >  To
> > fix, I think we should modify postgresPlanDirectModify() so that it
> > clears the async-capable flag if it is set.  Attached is a patch for
> > that.  Maybe I am missing something, though.
>
> I see that your patch is to disable asynchronous execution in
> ForeignScan nodes responsible for direct update/delete, but why not do
> the same for other ForeignScan nodes too?

I just thought it would be better to execute other ForeignScan nodes
asynchronously for performance, if they are async-capable.

> Or the other way around --
> is it because fixing the crash that occurs in the former's case would
> be a significant undertaking for little gain?

Yeah, I think it would be a good idea to support "Async Foreign
Delete" in the former's case.  And actually, I tried to do so, but I
didn't, because it seemed to take time.  I might be missing something,
though.

Thanks!

Best regards,
Etsuro Fujita

Re: Inherited UPDATE/DELETE vs async execution

Amit Langote <amitlangote09@gmail.com> — 2021-05-11T12:53:24Z

Fujita-san,

On Tue, May 11, 2021 at 5:56 PM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> On Mon, May 10, 2021 at 9:21 PM Amit Langote <amitlangote09@gmail.com> wrote:
> > On Sat, May 8, 2021 at 1:21 AM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> > > I noticed this while working on the
> > > EXPLAIN-ANALYZE-for-async-capable-nodes issue:
> > >
> > > EXPLAIN (VERBOSE, COSTS OFF)
> > > DELETE FROM async_pt;
> > >                            QUERY PLAN
> > > ----------------------------------------------------------------
> > >  Delete on public.async_pt
> > >    Foreign Delete on public.async_p1 async_pt_1
> > >    Foreign Delete on public.async_p2 async_pt_2
> > >    Delete on public.async_p3 async_pt_3
> > >    ->  Append
> > >          ->  Async Foreign Delete on public.async_p1 async_pt_1
> > >                Remote SQL: DELETE FROM public.base_tbl1
> > >          ->  Async Foreign Delete on public.async_p2 async_pt_2
> > >                Remote SQL: DELETE FROM public.base_tbl2
> > >          ->  Seq Scan on public.async_p3 async_pt_3
> > >                Output: async_pt_3.tableoid, async_pt_3.ctid
> > > (11 rows)
> > >
> > > DELETE FROM async_pt;
> > > server closed the connection unexpectedly
> > >     This probably means the server terminated abnormally
> > >     before or while processing the request.
> > > connection to server was lost
> > >
> > > The cause for this would be that direct-update plans are mistakenly
> > > treated as async-capable ones, as shown in the EXPLAIN output.
> >
> > I guess that happens because the ForeignScan nodes responsible for
> > scanning or direct-updating/deleting from child foreign tables appear
> > under an Append as of 86dc90056, whereas before they would appear as
> > child plans of a ModifyTable node.  IIUC, it's the Append that causes
> > the async_capable flag to be set in those ForeignScan nodes.
>
> That's right.
>
> The inherited update/delete work is great!  Thanks for that!

Thanks.

> > >  To
> > > fix, I think we should modify postgresPlanDirectModify() so that it
> > > clears the async-capable flag if it is set.  Attached is a patch for
> > > that.  Maybe I am missing something, though.
> >
> > I see that your patch is to disable asynchronous execution in
> > ForeignScan nodes responsible for direct update/delete, but why not do
> > the same for other ForeignScan nodes too?
>
> I just thought it would be better to execute other ForeignScan nodes
> asynchronously for performance, if they are async-capable.

Okay, so I take it that making these ForeignScan nodes (that only
fetch the data) asynchronous doesn't interfere with update/delete
subsequently being performed over presumably the same connection to
the remote server.

> > Or the other way around --
> > is it because fixing the crash that occurs in the former's case would
> > be a significant undertaking for little gain?
>
> Yeah, I think it would be a good idea to support "Async Foreign
> Delete" in the former's case.  And actually, I tried to do so, but I
> didn't, because it seemed to take time.

Ah I see.  I guess it makes sense to prevent such cases in v14 as your
patch does, and revisit this in the future.
--
Amit Langote
EDB: http://www.enterprisedb.com

Re: Inherited UPDATE/DELETE vs async execution

Etsuro Fujita <etsuro.fujita@gmail.com> — 2021-05-12T09:44:56Z

Amit-san,

On Tue, May 11, 2021 at 9:53 PM Amit Langote <amitlangote09@gmail.com> wrote:
> On Tue, May 11, 2021 at 5:56 PM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> > On Mon, May 10, 2021 at 9:21 PM Amit Langote <amitlangote09@gmail.com> wrote:
> > > On Sat, May 8, 2021 at 1:21 AM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:

> > > >  To
> > > > fix, I think we should modify postgresPlanDirectModify() so that it
> > > > clears the async-capable flag if it is set.  Attached is a patch for
> > > > that.  Maybe I am missing something, though.
> > >
> > > I see that your patch is to disable asynchronous execution in
> > > ForeignScan nodes responsible for direct update/delete, but why not do
> > > the same for other ForeignScan nodes too?
> >
> > I just thought it would be better to execute other ForeignScan nodes
> > asynchronously for performance, if they are async-capable.
>
> Okay, so I take it that making these ForeignScan nodes (that only
> fetch the data) asynchronous doesn't interfere with update/delete
> subsequently being performed over presumably the same connection to
> the remote server.

Good point!  I don't think it would interfere with the update/delete,
because in that case postgres_fdw would actually perform the
update/delete and the asynchronous foreign scans serially rather than
concurrently.  (They wouldn't be perfomed in parallel unless they use
different connections, in other words.)

> > > Or the other way around --
> > > is it because fixing the crash that occurs in the former's case would
> > > be a significant undertaking for little gain?
> >
> > Yeah, I think it would be a good idea to support "Async Foreign
> > Delete" in the former's case.  And actually, I tried to do so, but I
> > didn't, because it seemed to take time.
>
> Ah I see.  I guess it makes sense to prevent such cases in v14 as your
> patch does, and revisit this in the future.

+1

Here is a rebased version of the patch.  I'm planning to apply this tommorow.

Thanks for the comment!

Best regards,
Etsuro Fujita

Re: Inherited UPDATE/DELETE vs async execution

Amit Langote <amitlangote09@gmail.com> — 2021-05-13T06:32:04Z

On Wed, May 12, 2021 at 6:45 PM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> On Tue, May 11, 2021 at 9:53 PM Amit Langote <amitlangote09@gmail.com> wrote:
> > Okay, so I take it that making these ForeignScan nodes (that only
> > fetch the data) asynchronous doesn't interfere with update/delete
> > subsequently being performed over presumably the same connection to
> > the remote server.
>
> Good point!  I don't think it would interfere with the update/delete,
> because in that case postgres_fdw would actually perform the
> update/delete and the asynchronous foreign scans serially rather than
> concurrently.  (They wouldn't be perfomed in parallel unless they use
> different connections, in other words.)

I see, that makes sense.

> > > > Or the other way around --
> > > > is it because fixing the crash that occurs in the former's case would
> > > > be a significant undertaking for little gain?
> > >
> > > Yeah, I think it would be a good idea to support "Async Foreign
> > > Delete" in the former's case.  And actually, I tried to do so, but I
> > > didn't, because it seemed to take time.
> >
> > Ah I see.  I guess it makes sense to prevent such cases in v14 as your
> > patch does, and revisit this in the future.
>
> +1
>
> Here is a rebased version of the patch.  I'm planning to apply this tommorow.

+   /*
+    * Finally, unset the async-capable flag if it is set.
+    */

Would it make sense to expand here even just a bit on why we must do this?

-- 
Amit Langote
EDB: http://www.enterprisedb.com

Re: Inherited UPDATE/DELETE vs async execution

Etsuro Fujita <etsuro.fujita@gmail.com> — 2021-05-13T08:00:22Z

On Thu, May 13, 2021 at 3:32 PM Amit Langote <amitlangote09@gmail.com> wrote:
> On Wed, May 12, 2021 at 6:45 PM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> > Here is a rebased version of the patch.  I'm planning to apply this tommorow.
>
> +   /*
> +    * Finally, unset the async-capable flag if it is set.
> +    */
>
> Would it make sense to expand here even just a bit on why we must do this?

+1  How about something like this?

"Finally, unset the async-capable flag if it is set, as we currently
don't support asynchronous execution of direct modifications."

Best regards,
Etsuro Fujita

Re: Inherited UPDATE/DELETE vs async execution

Etsuro Fujita <etsuro.fujita@gmail.com> — 2021-05-13T11:10:52Z

On Thu, May 13, 2021 at 5:00 PM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
>
> On Thu, May 13, 2021 at 3:32 PM Amit Langote <amitlangote09@gmail.com> wrote:
> > On Wed, May 12, 2021 at 6:45 PM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> > > Here is a rebased version of the patch.  I'm planning to apply this tommorow.
> >
> > +   /*
> > +    * Finally, unset the async-capable flag if it is set.
> > +    */
> >
> > Would it make sense to expand here even just a bit on why we must do this?
>
> +1  How about something like this?
>
> "Finally, unset the async-capable flag if it is set, as we currently
> don't support asynchronous execution of direct modifications."

Pushed after modifying the comment as such.  I think we could improve
it later.  :-)

Thanks for the comment!

Will close this in the open items list.

Best regards,
Etsuro Fujita

Re: Inherited UPDATE/DELETE vs async execution

Amit Langote <amitlangote09@gmail.com> — 2021-05-13T12:09:33Z

On Thu, May 13, 2021 at 8:10 PM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> On Thu, May 13, 2021 at 5:00 PM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> > On Thu, May 13, 2021 at 3:32 PM Amit Langote <amitlangote09@gmail.com> wrote:
> > > On Wed, May 12, 2021 at 6:45 PM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> > > > Here is a rebased version of the patch.  I'm planning to apply this tommorow.
> > >
> > > +   /*
> > > +    * Finally, unset the async-capable flag if it is set.
> > > +    */
> > >
> > > Would it make sense to expand here even just a bit on why we must do this?
> >
> > +1  How about something like this?
> >
> > "Finally, unset the async-capable flag if it is set, as we currently
> > don't support asynchronous execution of direct modifications."
>
> Pushed after modifying the comment as such.  I think we could improve
> it later.  :-)

Looks good as pushed, thank you.

-- 
Amit Langote
EDB: http://www.enterprisedb.com