Thread

Re: [WIP] Allow pg_upgrade to copy segments of the same relfilenode in parallel

Jaime Casanova <jcasanov@systemguards.com.ec> — 2022-03-27T16:07:27Z

On Mon, Mar 21, 2022 at 05:34:31PM -0700, Andres Freund wrote:
> Hi,
> 
> On 2022-02-01 21:57:00 -0500, Jaime Casanova wrote:
> > This patch adds a new option (-J num, --jobs-per-disk=num) in 
> > pg_upgrade to speed up copy mode. This generates upto ${num} 
> > processes per tablespace to copy segments of the same relfilenode 
> > in parallel.
> > 
> > This can help when you have many multi gigabyte tables (each segment 
> > is 1GB by default) in different tablespaces (each tablespace in a 
> > different disk) and multiple processors.
> > 
> > In a customer's database (~20Tb) it went down from 6h to 4h 45min.
> > 
> > It lacks documentation and I need help with WIN32 part of it, I created
> > this new mail to put the patch on the next commitfest.
> 
> The patch currently fails on cfbot due to warnings, likely related due to the
> win32 issue: https://cirrus-ci.com/task/4566046517493760?logs=mingw_cross_warning#L388
> 
> As it's a new patch submitted to the last CF, hasn't gotten any review yet and
> misses some platform support, it seems like there's no chance it can make it
> into 15?
> 

Hi,

Because I have zero experience on the windows side of this, I will take
some time to complete that part.

Should we move this to the next commitfest (and make 16 the target for
this)?

-- 
Jaime Casanova
Director de Servicios Profesionales
SystemGuards - Consultores de PostgreSQL