Horton

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

Robert Haas <robertmhaas@gmail.com>

View thread View on postgr.es
From: Robert Haas <robertmhaas@gmail.com>
To: Peter Geoghegan <pg@bowt.ie>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>, Rushabh Lathia <rushabh.lathia@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Pg Hackers <pgsql-hackers@postgresql.org>, Corey Huinker <corey.huinker@gmail.com>
Date: 2018-01-18T18:17:35Z
Lists: pgsql-hackers 
On Wed, Jan 17, 2018 at 10:22 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> If you think it's worth the cycles, then I have no objection. I will
> point out that this means that everything that I say about
> ReindexIsProcessingIndex() no longer applies, because the relevant
> state will now be propagated. It doesn't need to be mentioned at all,
> and I don't even need to forbid builds on catalogs.
>
> Should I go ahead and restore builds on catalogs, and remove those
> comments, on the assumption that your patch will be committed before
> mine? Obviously parallel index builds on catalogs don't matter. OTOH,
> why not? Perhaps it's like the debate around HOT that took place over
> 10 years ago, where Tom insisted that HOT work with catalogs on
> general principle.

Yes, I think so.  If you (or someone else) can review that patch, I'll
go ahead and commit it, and then your patch can treat it as a solved
problem.  I'm not really worried about the cycles; the amount of
effort required here is surely very small compared to all of the other
things that have to be done when starting a parallel worker.

I'm not as dogmatic about the idea that everything must support system
catalogs or it's not worth doing as Tom is, but I do think it's better
if it can be done that way with reasonable effort.  When each new
feature comes with a set of unsupported corner cases, it becomes hard
for users to understand what will and will not actually work.  Now,
really big features like parallel query or partitioning or logical
replication generally do need to exclude some things in v1 or you can
never finish the project, but in this case plugging the gap seems
quite feasible.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Commits

  1. Support parallel btree index builds.

  2. Report an ERROR if a parallel worker fails to start properly.

  3. Transfer state pertaining to pending REINDEX operations to workers.

  4. Add a barrier primitive for synchronizing backends.

  5. Allow DML commands that create tables to use parallel query.

  6. Refactor GetOldestXmin() to use flags

  7. Fix regression in parallel planning against inheritance tables.

  8. Don't create "holes" in BufFiles, in the new logtape code.

  9. Simplify the code for logical tape read buffers.

  10. Fix excessive memory consumption in the new sort pre-reading code.

  11. Implement binary heap replace-top operation in a smarter way.

  12. Cosmetic code cleanup in commands/extension.c.

  13. Speed up planner's scanning for parallel-query hazards.

  14. Read from the same worker repeatedly until it returns no tuple.

  15. Improve tuplesort.c to support variable merge order. The original coding