Horton

Re: Parallel Append implementation

Robert Haas <robertmhaas@gmail.com>

View thread View on postgr.es
From: Robert Haas <robertmhaas@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Amit Khandekar <amitdkhan.pg@gmail.com>, Rafia Sabih <rafia.sabih@enterprisedb.com>, Andres Freund <andres@anarazel.de>, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2017-09-14T16:11:05Z
Lists: pgsql-hackers 
On Mon, Sep 11, 2017 at 9:25 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> I think the patch stores only non-partial paths in decreasing order,
> what if partial paths having more costs follows those paths?

The general picture here is that we don't want the leader to get stuck
inside some long-running operation because then it won't be available
to read tuples from the workers.  On the other hand, we don't want to
just have the leader do no work because that might be slow.  And in
most cast cases, the leader will be the first participant to arrive at
the Append node, because of the worker startup time.  So the idea is
that the workers should pick expensive things first, and the leader
should pick cheap things first.  This may not always work out
perfectly and certainly the details of the algorithm may need some
refinement, but I think the basic concept is good.  Of course, that
may be because I proposed it...

Note that there's a big difference between the leader picking a
partial path and the leader picking a non-partial path.  If the leader
picks a partial path, it isn't committed to executing that path to
completion.  Other workers can help.  If the leader picks a
non-partial path, though, the workers are locked out of that path,
because a single process must run it all the way through.  So the
leader should avoid choosing a non-partial path unless there are no
partial paths remaining.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Commits

  1. Update parallel.sgml for Parallel Append

  2. Support Parallel Append plan nodes.

  3. Remove BufFile's isTemp flag.

  4. Improve comments for parallel executor estimation functions.

  5. Separate reinitialization of shared parallel-scan state from ExecReScan.

  6. Eat XIDs more efficiently in recovery TAP test.

  7. Avoid syntax error on platforms that have neither LOCALE_T nor ICU.

  8. Preparatory refactoring for parallel merge join support.