Horton
Menu

Re: Parallel CREATE INDEX for GIN indexes

Tomas Vondra <tomas@vondra.me>

View thread
From: Tomas Vondra <tomas@vondra.me>
To: Matthias van de Meent <boekewurm+postgres@gmail.com>
Cc: Kirill Reshke <reshkekirill@gmail.com>, Michael Paquier <michael@paquier.xyz>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2025-02-25T15:49:03Z
Lists: pgsql-hackers

Attachments

One more patch version / rebase. I've been planning to get 0001
committed, but I realized there's one more loose end - progress reporting.

I could have committed it without it, I guess, but Matthias actually
mentioned this a couple days ago so I took a stab at it. The build goes
through these 5 build stages (on top of "INITIALIZE"):

  PROGRESS_GIN_PHASE_INDEXBUILD_TABLESCAN
  PROGRESS_GIN_PHASE_PERFORMSORT_1
  PROGRESS_GIN_PHASE_MERGE_1
  PROGRESS_GIN_PHASE_PERFORMSORT_2
  PROGRESS_GIN_PHASE_MERGE_2

The phases up to PROGRESS_GIN_PHASE_MERGE_1 happen in workers, i.e. it
ends with workers feeding the sorted/merged data into the shared
tuplesort. The last two phases are in the leader, which merges the data
and actually inserts it into the GIN index.

The "parallel" part has the blocks_done/blocks_total showing progress,
per the parallel scan. The "leader" phases use tuples_done/tuples_total,
where "tuple" is the GIN tuple produced by workers (each worker reports
the number of "tuples" it writes into the shared tuplesort, the leader
then tracks how many it processed).

I think this works pretty nicely. I'm not entirely sure we need all the
phases, maybe it'd be fine to have the sort+merge as a single phase? Or
maybe there should be one extra "sort" phase? Workers do two sorts,
first on their "private" tuplesort, then on the "shared" one.

What annoys me a little bit is that we only see those stages if the
leader participates as a worker. With parallel_leader_participation=off
none of this is visible anyway (we still see the blocks from the scan).


regards

-- 
Tomas Vondra

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Exercise parallel GIN builds in regression tests

  2. Lookup the correct ordering for parallel GIN builds

  3. Fix memory leak in _gin_parallel_merge

  4. Enforce memory limit during parallel GIN builds

  5. Compress TID lists when writing GIN tuples to disk

  6. Allow parallel CREATE INDEX for GIN indexes