Horton

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

Thomas Munro <thomas.munro@enterprisedb.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@enterprisedb.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Peter Geoghegan <pg@bowt.ie>, Rushabh Lathia <rushabh.lathia@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Pg Hackers <pgsql-hackers@postgresql.org>, Corey Huinker <corey.huinker@gmail.com>
Date: 2018-01-19T21:27:03Z
Lists: pgsql-hackers 
On Sat, Jan 20, 2018 at 6:32 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Jan 19, 2018 at 12:16 PM, Peter Geoghegan <pg@bowt.ie> wrote:
>> Clarity on what I should do about parallel_leader_participation in the
>> next revision would be useful at this point. You seem to either want
>> me to remove it from consideration entirely, or to remove the code
>> that specifically disallows a "degenerate parallel CREATE INDEX". I
>> need a final answer on that.
>
> Right.  I do think that we should do one of those things, and I lean
> towards removing it entirely, but I'm not entirely sure.    Rather
> than making an executive decision immediately, I'd like to wait a few
> days to give others a chance to comment. I am hoping that we might get
> some other opinions, especially from Thomas who implemented
> parallel_leader_participation, or maybe Amit who has been reviewing
> recently, or anyone else who is paying attention to this thread.

Well, I see parallel_leader_participation as having these reasons to exist:

1.  Gather could in rare circumstances not run the plan in the leader.
This can hide bugs.  It's good to be able to force that behaviour for
testing.

2.  Plans that tie up the leader process for a long time cause the
tuple queues to block, which reduces parallelism.  I speculate that
some people might want to turn that off in production, but at the very
least it seems useful for certain kinds of performance testing to be
able to remove this complication from the picture.

3.  The planner's estimations of parallel leader contribution are
somewhat bogus, especially if the startup cost is high.  It's useful
to be able to remove that problem from the picture sometimes, at least
for testing and development work.

Parallel CREATE INDEX doesn't have any of those problems.  The only
reason I can see for it to respect parallel_leader_participation = off
is for consistency with Gather.  If someone decides to run their
cluster with that setting, then it's slightly odd if CREATE INDEX
scans and sorts with one extra process, but it doesn't seem like a big
deal.

I vote for removing the GUC from consideration for now (ie always use
the leader), and revisiting the question again later when we have more
experience or if the parallel degree logic becomes more sophisticated
in future.

-- 
Thomas Munro
http://www.enterprisedb.com

Commits

  1. Support parallel btree index builds.

  2. Report an ERROR if a parallel worker fails to start properly.

  3. Transfer state pertaining to pending REINDEX operations to workers.

  4. Add a barrier primitive for synchronizing backends.

  5. Allow DML commands that create tables to use parallel query.

  6. Refactor GetOldestXmin() to use flags

  7. Fix regression in parallel planning against inheritance tables.

  8. Don't create "holes" in BufFiles, in the new logtape code.

  9. Simplify the code for logical tape read buffers.

  10. Fix excessive memory consumption in the new sort pre-reading code.

  11. Implement binary heap replace-top operation in a smarter way.

  12. Cosmetic code cleanup in commands/extension.c.

  13. Speed up planner's scanning for parallel-query hazards.

  14. Read from the same worker repeatedly until it returns no tuple.

  15. Improve tuplesort.c to support variable merge order. The original coding