Horton

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

Peter Geoghegan <pg@bowt.ie>

View thread View on postgr.es
From: Peter Geoghegan <pg@bowt.ie>
To: Thomas Munro <thomas.munro@enterprisedb.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Rushabh Lathia <rushabh.lathia@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Pg Hackers <pgsql-hackers@postgresql.org>, Corey Huinker <corey.huinker@gmail.com>
Date: 2018-01-25T02:05:46Z
Lists: pgsql-hackers 
On Wed, Jan 24, 2018 at 5:31 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> Here's a version that works, and a minimal repro test module thing.
> Without 0003 applied, it hangs.

I can confirm that this version does in fact fix the problem with
parallel CREATE INDEX hanging in the event of (simulated) worker
fork() failure. And, it seems to have at least one tiny advantage over
the other approaches I was talking about that you didn't mention,
which is that we never have to wait until the leader stops
participating as a worker before an error is raised. IOW, either the
whole parallel CREATE INDEX operation throws an error at an early
point in the CREATE INDEX, or the CREATE INDEX completely succeeds.

Obviously, the other, stated advantage is more relevant: *everyone*
automatically doesn't have to worry about nworkers_launched being
inaccurate this way, including code that gets away with this today
only due to using a tuple queue, such as nodeGather.c, but may not
always get away with it in the future.

I've run out of time to assess what you've done here in any real
depth. For now, I will say that this approach seems interesting to me.
I'll take a closer look tomorrow.

-- 
Peter Geoghegan

Commits

  1. Support parallel btree index builds.

  2. Report an ERROR if a parallel worker fails to start properly.

  3. Transfer state pertaining to pending REINDEX operations to workers.

  4. Add a barrier primitive for synchronizing backends.

  5. Allow DML commands that create tables to use parallel query.

  6. Refactor GetOldestXmin() to use flags

  7. Fix regression in parallel planning against inheritance tables.

  8. Don't create "holes" in BufFiles, in the new logtape code.

  9. Simplify the code for logical tape read buffers.

  10. Fix excessive memory consumption in the new sort pre-reading code.

  11. Implement binary heap replace-top operation in a smarter way.

  12. Cosmetic code cleanup in commands/extension.c.

  13. Speed up planner's scanning for parallel-query hazards.

  14. Read from the same worker repeatedly until it returns no tuple.

  15. Improve tuplesort.c to support variable merge order. The original coding