Re: Parallel CREATE INDEX for GIN indexes

Tomas Vondra <tomas@vondra.me>

View thread

From: Tomas Vondra <tomas@vondra.me>

To: Michael Paquier <michael@paquier.xyz>, Tomas Vondra <tomas.vondra@enterprisedb.com>

Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2024-10-08T12:06:21Z

Lists: pgsql-hackers

Attachments

v20241008-0001-Allow-parallel-create-for-GIN-indexes.patch (text/x-patch) patch v20241008-0001
v20241008-0002-Use-mergesort-in-the-leader-process.patch (text/x-patch) patch v20241008-0002
v20241008-0003-Remove-the-explicit-pg_qsort-in-workers.patch (text/x-patch) patch v20241008-0003
v20241008-0004-Compress-TID-lists-before-writing-tuples-t.patch (text/x-patch) patch v20241008-0004
v20241008-0005-Collect-and-print-compression-stats.patch (text/x-patch) patch v20241008-0005
v20241008-0006-Enforce-memory-limit-when-combining-tuples.patch (text/x-patch) patch v20241008-0006
v20241008-0007-Detect-wrap-around-in-parallel-callback.patch (text/x-patch) patch v20241008-0007
v20241008-0008-Use-a-single-GIN-tuplesort.patch (text/x-patch) patch v20241008-0008
v20241008-0009-Reduce-the-size-of-GinTuple-by-12-bytes.patch (text/x-patch) patch v20241008-0009
v20241008-0010-WIP-parallel-inserts-into-GIN-index.patch (text/x-patch) patch v20241008-0010

On 10/8/24 04:03, Michael Paquier wrote:
>
> _gin_parallel_build_main() is introduced in 0001.  Please make sure to
> pass down a query ID.

Thanks for the ping. Here's an updated patch doing that, and also fixing
a couple whitespace issues. No other changes, but I plan to get back to
this patch soon - before the next CF.

regards

-- 
Tomas Vondra

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Exercise parallel GIN builds in regression tests
- f6e5d21bf73f 19 (unreleased) landed
- 32593394ee43 18.2 landed
Lookup the correct ordering for parallel GIN builds
- db14dcdec69d 19 (unreleased) landed
- eee71a66cc86 18.2 landed
Fix memory leak in _gin_parallel_merge
- 1681a70df3d6 18.0 landed
Enforce memory limit during parallel GIN builds
- b229c1016477 18.0 landed
Compress TID lists when writing GIN tuples to disk
- 0b2a45a5d1f2 18.0 landed
Allow parallel CREATE INDEX for GIN indexes
- 8492feb98f6d 18.0 landed