Horton

Re: Parallel Append implementation

Robert Haas <robertmhaas@gmail.com>

View thread View on postgr.es
From: Robert Haas <robertmhaas@gmail.com>
To: Ashutosh Bapat <ashutosh.bapat@enterprisedb.com>
Cc: Amit Khandekar <amitdkhan.pg@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2017-02-19T09:03:11Z
Lists: pgsql-hackers 
On Fri, Feb 17, 2017 at 11:44 AM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:
> That's true for a partitioned table, but not necessarily for every
> append relation. Amit's patch is generic for all append relations. If
> the child plans are joins or subquery segments of set operations, I
> doubt if the same logic works. It may be better if we throw as many
> workers (or some function "summing" those up) as specified by those
> subplans. I guess, we have to use different logic for append relations
> which are base relations and append relations which are not base
> relations.

Well, I for one do not believe that if somebody writes a UNION ALL
with 100 branches, they should get 100 (or 99) workers.  Generally
speaking, the sweet spot for parallel workers on queries we've tested
so far has been between 1 and 4.  It's straining credulity to believe
that the number that's correct for parallel append is more than an
order of magnitude larger.  Since increasing resource commitment by
the logarithm of the problem size has worked reasonably well for table
scans, I believe we should pursue a similar approach here.  I'm
willing to negotiate on the details of what the formula I looked like,
but I'm not going to commit something that lets an Append relation try
to grab massively more resources than we'd use for some other plan
shape.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Commits

  1. Update parallel.sgml for Parallel Append

  2. Support Parallel Append plan nodes.

  3. Remove BufFile's isTemp flag.

  4. Improve comments for parallel executor estimation functions.

  5. Separate reinitialization of shared parallel-scan state from ExecReScan.

  6. Eat XIDs more efficiently in recovery TAP test.

  7. Avoid syntax error on platforms that have neither LOCALE_T nor ICU.

  8. Preparatory refactoring for parallel merge join support.