Horton

Re: why not parallel seq scan for slow functions

Jeff Janes <jeff.janes@gmail.com>

View thread View on postgr.es
From: Jeff Janes <jeff.janes@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>, Amit Khandekar <amitdkhan.pg@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2017-09-20T21:05:35Z
Lists: pgsql-hackers 
On Tue, Sep 19, 2017 at 9:15 PM, Amit Kapila <amit.kapila16@gmail.com>
wrote:

> On Wed, Sep 20, 2017 at 3:05 AM, Jeff Janes <jeff.janes@gmail.com> wrote:
> > On Tue, Sep 19, 2017 at 1:17 PM, Thomas Munro
> > <thomas.munro@enterprisedb.com> wrote:
> >>
> >> On Thu, Sep 14, 2017 at 3:19 PM, Amit Kapila <amit.kapila16@gmail.com>
> >> wrote:
> >> > The attached patch fixes both the review comments as discussed above.
> >
> >
> > that should be fixed by turning costs on the explain, as is the
> tradition.
> >
>
> Right.  BTW, did you get a chance to run the original test (for which
> you have reported the problem) with this patch?
>

Yes, this patch makes it use a parallel scan, with great improvement.  No
more having to \copy the data out, then run GNU split, then run my perl or
python program with GNU parallel on each file.  Instead I just have to put
a pl/perl wrapper around the function.

(next up, how to put a "create temp table alsdkfjaslfdj as" in front of it
and keep it running in parallel)

Thanks,

Jeff

Commits

  1. Rewrite the code that applies scan/join targets to paths.

  2. Teach create_projection_plan to omit projection where possible.

  3. Make the upper part of the planner work by generating and comparing Paths.