Horton

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

Peter Geoghegan <pg@bowt.ie>

View thread View on postgr.es
From: Peter Geoghegan <pg@bowt.ie>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>, Robert Haas <robertmhaas@gmail.com>, Rushabh Lathia <rushabh.lathia@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Pg Hackers <pgsql-hackers@postgresql.org>, Corey Huinker <corey.huinker@gmail.com>
Date: 2018-01-20T03:03:22Z
Lists: pgsql-hackers 
On Fri, Jan 19, 2018 at 6:52 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> Or reverse is also possible which means the workers won't get chance
> to run the plan in which case we can use parallel_leader_participation
> = off to test workers behavior.  As said before, I see only that as
> the reason to keep parallel_leader_participation in this patch.  If we
> decide to do that way, then I think we should remove the code that
> specifically disallows a "degenerate parallel CREATE INDEX" as that
> seems to be confusing.   If we go this way, then I think we should use
> the wording suggested by Robert in one of its email [1] to describe
> the usage of parallel_leader_participation.

I agree that parallel_leader_participation is only useful for testing
in the context of parallel CREATE INDEX. My concern with allowing a
"degenerate parallel CREATE INDEX" to go ahead is that
parallel_leader_participation generally isn't just intended for
testing by hackers (if it was, then I wouldn't care). But I'm now more
than willing to let this go.

> BTW, is there any other way for "parallel create index" to force that
> the work is done by workers?  I am insisting on having something which
> can test the code path in workers because we have found quite a few
> bugs using that idea.

I agree that this is essential (more so than supporting
parallel_leader_participation). You can use the parallel_workers table
storage parameter for this. When the storage param has been set, we
don't care about the amount of memory available to each worker. You
can stress-test the implementation as needed. (The storage param does
care about max_parallel_maintenance_workers, but you can set that as
high as you like.)

-- 
Peter Geoghegan

Commits

  1. Support parallel btree index builds.

  2. Report an ERROR if a parallel worker fails to start properly.

  3. Transfer state pertaining to pending REINDEX operations to workers.

  4. Add a barrier primitive for synchronizing backends.

  5. Allow DML commands that create tables to use parallel query.

  6. Refactor GetOldestXmin() to use flags

  7. Fix regression in parallel planning against inheritance tables.

  8. Don't create "holes" in BufFiles, in the new logtape code.

  9. Simplify the code for logical tape read buffers.

  10. Fix excessive memory consumption in the new sort pre-reading code.

  11. Implement binary heap replace-top operation in a smarter way.

  12. Cosmetic code cleanup in commands/extension.c.

  13. Speed up planner's scanning for parallel-query hazards.

  14. Read from the same worker repeatedly until it returns no tuple.

  15. Improve tuplesort.c to support variable merge order. The original coding