Horton

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

Peter Geoghegan <pg@bowt.ie>

View thread View on postgr.es
From: Peter Geoghegan <pg@bowt.ie>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>, Rushabh Lathia <rushabh.lathia@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Pg Hackers <pgsql-hackers@postgresql.org>, Corey Huinker <corey.huinker@gmail.com>
Date: 2018-01-10T22:42:31Z
Lists: pgsql-hackers 
On Wed, Jan 10, 2018 at 2:21 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> I share your general feelings on all of this, but I really don't know
>> what to do about it. Which of these alternatives is the least worst,
>> all things considered?
>
> Let's get the patch committed without any explicit way of forcing the
> number of workers and then think about adding that later.

It could be argued that you need some way of forcing low memory in
workers with any committed version. So while this sounds reasonable,
it might not be compatible with throwing out what I've done with
force_parallel_mode up-front, before you commit anything. What do you
think?

> It will be good if you and Rushabh can agree on who will produce the
> next version of this patch, and also if I have some idea when that
> version should be expected.

I'll take it.

> On another point, we will need to agree
> on how this should be credited in an eventual commit message.  I do
> not agree with adding Heikki as an author unless he contributed code,
> but we can credit him in some other way, like "Thanks are also due to
> Heikki Linnakangas for significant improvements to X, Y, and Z that
> made this patch possible."

I agree that I should have been more nuanced with this. Here's what I intended:

Heikki is not the author of any of the code in the final commit, but
he is morally a (secondary) author of the feature as a whole, and
should be credited as such within the final release notes. This is
justified by the history here, which is that he was involved with the
patch fairly early on, and did some work that was particularly
important to the feature, that almost certainly would not otherwise
have happened. Sure, it helped the serial case too, but much less so.
That's really not why he did it.

> I assume the author credit will be "Peter
> Geoghegan, Rushabh Lathia" in that order, but let me know if anyone
> thinks that isn't the right idea.

"Peter Geoghegan, Rushabh Lathia" seems right. Thomas did write a very
small amount of the actual code, but I think it was more of a review
thing (he is already credited as a reviewer).

-- 
Peter Geoghegan

Commits

  1. Support parallel btree index builds.

  2. Report an ERROR if a parallel worker fails to start properly.

  3. Transfer state pertaining to pending REINDEX operations to workers.

  4. Add a barrier primitive for synchronizing backends.

  5. Allow DML commands that create tables to use parallel query.

  6. Refactor GetOldestXmin() to use flags

  7. Fix regression in parallel planning against inheritance tables.

  8. Don't create "holes" in BufFiles, in the new logtape code.

  9. Simplify the code for logical tape read buffers.

  10. Fix excessive memory consumption in the new sort pre-reading code.

  11. Implement binary heap replace-top operation in a smarter way.

  12. Cosmetic code cleanup in commands/extension.c.

  13. Speed up planner's scanning for parallel-query hazards.

  14. Read from the same worker repeatedly until it returns no tuple.

  15. Improve tuplesort.c to support variable merge order. The original coding