Horton

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

Peter Geoghegan <pg@bowt.ie>

View thread View on postgr.es
From: Peter Geoghegan <pg@bowt.ie>
To: Rushabh Lathia <rushabh.lathia@gmail.com>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>, Robert Haas <robertmhaas@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Pg Hackers <pgsql-hackers@postgresql.org>, Corey Huinker <corey.huinker@gmail.com>
Date: 2017-11-14T18:01:05Z
Lists: pgsql-hackers 
On Tue, Nov 14, 2017 at 1:41 AM, Rushabh Lathia
<rushabh.lathia@gmail.com> wrote:
> Thanks Peter and Thomas for the review comments.

No problem. More feedback:

* I don't really see much need for this:

+ elog(LOG, "Worker for create index %d", parallel_workers);

You can just use trace_sort, and observe the actual behavior of the
sort that way.

* As I said before, you should remove the header comments within nbtsort.c.

* This should just say "write routines":

+ * This is why write/recycle routines don't need to know about offsets at
+ * all.

* You didn't point out the randomAccess restriction in tuplesort.h.

* I can't remember why I added the Valgrind suppression at this point.
I'd remove it until the reason becomes clear, which may never happen.
The regression tests should still pass without Valgrind warnings.

* You can add back comments removed from above LogicalTapeTell(). I
made these changes because it looked like we should close out the
possibility of doing a tell during the write phase, as unified tapes
actually would make that hard (no one does what it describes anyway).
But now, unified tapes are a distinct case to frozen tapes in a way
that they weren't before, so there is no need to make it impossible.

I also think you should replace "Assert(lt->frozen)" with
"Assert(lt->offsetBlockNumber == 0L)", for the same reason.

-- 
Peter Geoghegan

Commits

  1. Support parallel btree index builds.

  2. Report an ERROR if a parallel worker fails to start properly.

  3. Transfer state pertaining to pending REINDEX operations to workers.

  4. Add a barrier primitive for synchronizing backends.

  5. Allow DML commands that create tables to use parallel query.

  6. Refactor GetOldestXmin() to use flags

  7. Fix regression in parallel planning against inheritance tables.

  8. Don't create "holes" in BufFiles, in the new logtape code.

  9. Simplify the code for logical tape read buffers.

  10. Fix excessive memory consumption in the new sort pre-reading code.

  11. Implement binary heap replace-top operation in a smarter way.

  12. Cosmetic code cleanup in commands/extension.c.

  13. Speed up planner's scanning for parallel-query hazards.

  14. Read from the same worker repeatedly until it returns no tuple.

  15. Improve tuplesort.c to support variable merge order. The original coding