Horton

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

Peter Geoghegan <pg@bowt.ie>

View thread View on postgr.es
From: Peter Geoghegan <pg@bowt.ie>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Thomas Munro <thomas.munro@enterprisedb.com>, Rushabh Lathia <rushabh.lathia@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Pg Hackers <pgsql-hackers@postgresql.org>, Corey Huinker <corey.huinker@gmail.com>
Date: 2018-01-18T03:28:05Z
Lists: pgsql-hackers 
On Wed, Jan 17, 2018 at 6:20 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> I had forgotten about the previous discussion.  The sketch in my
> previous email supposed that we would use dynamic barriers since the
> whole point, after all, is to handle the fact that we don't know how
> many participants will really show up.  Thomas's idea seems to be that
> the leader will initialize the barrier based on the anticipated number
> of participants and then tell it to forget about the participants that
> don't materialize.  Of course, that would require that the leader
> somehow figure out how many participants didn't show up so that it can
> deduct then from the counter in the barrier.  And how is it going to
> do that?

I don't know; Thomas?

> It's true that the leader will know the value of nworkers_launched,
> but as the comment in LaunchParallelWorkers() says: "The caller must
> be able to tolerate ending up with fewer workers than expected, so
> there is no need to throw an error here if registration fails.  It
> wouldn't help much anyway, because registering the worker in no way
> guarantees that it will start up and initialize successfully."  So it
> seems to me that a much better plan than having the leader try to
> figure out how many workers failed to launch would be to just keep a
> count of how many workers did in fact launch.

> So my position (at least until Thomas or Andres shows up and tells me
> why I'm wrong) is that you can use the Barrier API just as it is
> without any yak-shaving, just by following the sketch I set out
> before.  The additional API I proposed in that sketch isn't really
> required, although it might be more efficient.  But it doesn't really
> matter: if that comes along later, it will be trivial to adjust the
> code to take advantage of it.

Okay. I'll work on adopting dynamic barriers in the way you described.
I just wanted to make sure that we're all on the same page about what
that looks like.

-- 
Peter Geoghegan

Commits

  1. Support parallel btree index builds.

  2. Report an ERROR if a parallel worker fails to start properly.

  3. Transfer state pertaining to pending REINDEX operations to workers.

  4. Add a barrier primitive for synchronizing backends.

  5. Allow DML commands that create tables to use parallel query.

  6. Refactor GetOldestXmin() to use flags

  7. Fix regression in parallel planning against inheritance tables.

  8. Don't create "holes" in BufFiles, in the new logtape code.

  9. Simplify the code for logical tape read buffers.

  10. Fix excessive memory consumption in the new sort pre-reading code.

  11. Implement binary heap replace-top operation in a smarter way.

  12. Cosmetic code cleanup in commands/extension.c.

  13. Speed up planner's scanning for parallel-query hazards.

  14. Read from the same worker repeatedly until it returns no tuple.

  15. Improve tuplesort.c to support variable merge order. The original coding