Re: Parallel tuplesort (for parallel B-Tree index creation)

Peter Geoghegan <pg@heroku.com>

From: Peter Geoghegan <pg@heroku.com>

To: Heikki Linnakangas <hlinnaka@iki.fi>

Cc: Claudio Freire <klaussfreire@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Pg Hackers <pgsql-hackers@postgresql.org>, Corey Huinker <corey.huinker@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>

Date: 2016-10-25T01:17:41Z

Lists: pgsql-hackers

Attachments

0003-Add-force_btree_randomaccess-GUC-for-testing.patch.gz (application/x-gzip) patch 0003
0002-Add-parallel-B-tree-index-build-sorting.patch.gz (application/x-gzip) patch 0002
0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz (application/x-gzip) patch 0001

On Fri, Oct 7, 2016 at 5:47 PM, Peter Geoghegan <pg@heroku.com> wrote:
> Work is still needed on:
>
> * Cost model. Should probably attempt to guess final index size, and
> derive calculation of number of workers from that. Also, I'm concerned
> that I haven't given enough thought to the low end, where with default
> settings most CREATE INDEX statements will use at least one parallel
> worker.
>
> * The whole way that I teach nbtsort.c to disallow catalog tables for
> parallel CREATE INDEX due to concerns about parallel safety is in need
> of expert review, preferably from Robert. It's complicated in a way
> that relies on things happening or not happening from a distance.
>
> * Heikki seems to want to change more about logtape.c, and its use of
> indirection blocks. That may evolve, but for now I can only target the
> master branch.
>
> * More extensive performance testing. I think that this V3 is probably
> the fastest version yet, what with Heikki's improvements, but I
> haven't really verified that.

While I haven't made progress on any of these open items, I should
still get a version out that applies cleanly on top of git tip --
commit b75f467b6eec0678452fd8d7f8d306e6df3a1076 caused the patch to
bitrot. I attach V4, which is a fairly mechanical rebase of V3, with
no notable behavioral changes or bug fixes.

-- 
Peter Geoghegan

Commits

Support parallel btree index builds.
- 9da0cc35284b 11.0 landed
Report an ERROR if a parallel worker fails to start properly.
- 2badb5afb89c 11.0 cited
Transfer state pertaining to pending REINDEX operations to workers.
- 29d58fd3adae 11.0 landed
Add a barrier primitive for synchronizing backends.
- 1145acc70deb 11.0 cited
Allow DML commands that create tables to use parallel query.
- e9baa5e9fa14 11.0 cited
Refactor GetOldestXmin() to use flags
- af4b1a0869bd 10.0 cited
Fix regression in parallel planning against inheritance tables.
- 2609e91fcf9d 10.0 cited
Don't create "holes" in BufFiles, in the new logtape code.
- 7ac4a389a7db 10.0 cited
Simplify the code for logical tape read buffers.
- b75f467b6eec 10.0 cited
Fix excessive memory consumption in the new sort pre-reading code.
- b56fb691b003 10.0 cited
Implement binary heap replace-top operation in a smarter way.
- 24598337c8d2 10.0 cited
Cosmetic code cleanup in commands/extension.c.
- 25794e841e5b 10.0 cited
Speed up planner's scanning for parallel-query hazards.
- da1c91631e35 10.0 cited
Read from the same worker repeatedly until it returns no tuple.
- bc7fcab5e36b 9.6.0 cited
Improve tuplesort.c to support variable merge order. The original coding
- df700e6b4019 8.2.0 cited