Horton

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. doc: Adjust note about pg_upgrade's --jobs option.

  2. Introduce framework for parallelizing various pg_upgrade tasks.

  1. doc: expand note about pg_upgrade's --jobs option

    Nathan Bossart <nathandbossart@gmail.com> — 2025-03-04T18:08:31Z

    Magnus noted to me off-list that the "et cetera" in the following sentence
in pg_upgrade's docs is doing quite a bit of heavy lifting:

	The --jobs option allows multiple CPU cores to be used for
	copying/linking of files, dumping and restoring database schemas in
	parallel, etc.; a good place to start is the maximum of the number of
	CPU cores and tablespaces.

I added the "et cetera" in commit 40e2e5e92b to cover the many follow-up
commits that parallelized various pg_upgrade tasks.  I was initially
worried that trying to list all the parallelized stuff would be too
verbose, but looking again, I think all the changes can be grouped into
"gathering cluster information" and "performing cluster checks."  The
attached patch replaces the "et cetera" with those two general categories.

-- 
nathan

  2. Re: doc: expand note about pg_upgrade's --jobs option

    Daniel Gustafsson <daniel@yesql.se> — 2025-03-05T10:00:34Z

    > On 4 Mar 2025, at 19:08, Nathan Bossart <nathandbossart@gmail.com> wrote:

> The attached patch replaces the "et cetera" with those two general categories.

LGTM.

--
Daniel Gustafsson

  3. Re: doc: expand note about pg_upgrade's --jobs option

    Magnus Hagander <magnus@hagander.net> — 2025-03-05T12:52:40Z

    On Wed, Mar 5, 2025 at 11:00 AM Daniel Gustafsson <daniel@yesql.se> wrote:

> > On 4 Mar 2025, at 19:08, Nathan Bossart <nathandbossart@gmail.com>
> wrote:
>
> > The attached patch replaces the "et cetera" with those two general
> categories.
>
> LGTM.
>

Another option that I think would also work is to just cut down the details
to just "The <option>--jobs</option> option allows multiple CPU cores to be
used".

I think this is also slightly confusing, but maybe that's a
non-native-english thing: "a good place to start is the maximum of the
number of  CPU cores and tablespaces.". Am I supposed to set it to
max(cpucores, ntablespaces) or to max(cpucores+ntablespaces)?

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ <http://www.hagander.net/>
 Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/>

  4. Re: doc: expand note about pg_upgrade's --jobs option

    Nathan Bossart <nathandbossart@gmail.com> — 2025-03-05T15:35:27Z

    On Wed, Mar 05, 2025 at 01:52:40PM +0100, Magnus Hagander wrote:
> Another option that I think would also work is to just cut down the details
> to just "The <option>--jobs</option> option allows multiple CPU cores to be
> used".

That's fine with me.  It's probably not particularly actionable
information, anyway.  If anything, IMHO we should make it clear to users
that the parallelization is per-database (except for file transfer, which
is per-tablespace).  If you've just got one big database in the default
tablespace, --jobs won't help.

> I think this is also slightly confusing, but maybe that's a
> non-native-english thing: "a good place to start is the maximum of the
> number of  CPU cores and tablespaces.". Am I supposed to set it to
> max(cpucores, ntablespaces) or to max(cpucores+ntablespaces)?

I've always read it to mean the former.  But I'm not sure that's great
advice.  If you have 8 cores and 100 tablespaces, does it make sense to use
--jobs=100?  Ordinarily, I'd suggest the number of cores as the starting
point.

-- 
nathan

  5. Re: doc: expand note about pg_upgrade's --jobs option

    Nathan Bossart <nathandbossart@gmail.com> — 2025-03-05T16:28:50Z

    On Wed, Mar 05, 2025 at 09:35:27AM -0600, Nathan Bossart wrote:
> On Wed, Mar 05, 2025 at 01:52:40PM +0100, Magnus Hagander wrote:
>> Another option that I think would also work is to just cut down the details
>> to just "The <option>--jobs</option> option allows multiple CPU cores to be
>> used".
> 
> That's fine with me.  It's probably not particularly actionable
> information, anyway.  If anything, IMHO we should make it clear to users
> that the parallelization is per-database (except for file transfer, which
> is per-tablespace).  If you've just got one big database in the default
> tablespace, --jobs won't help.
> 
>> I think this is also slightly confusing, but maybe that's a
>> non-native-english thing: "a good place to start is the maximum of the
>> number of  CPU cores and tablespaces.". Am I supposed to set it to
>> max(cpucores, ntablespaces) or to max(cpucores+ntablespaces)?
> 
> I've always read it to mean the former.  But I'm not sure that's great
> advice.  If you have 8 cores and 100 tablespaces, does it make sense to use
> --jobs=100?  Ordinarily, I'd suggest the number of cores as the starting
> point.

Here's another attempt at the patch based on the latest discussion.

-- 
nathan

  6. Re: doc: expand note about pg_upgrade's --jobs option

    Magnus Hagander <magnus@hagander.net> — 2025-03-08T00:43:52Z

    On Wed, Mar 5, 2025 at 5:28 PM Nathan Bossart <nathandbossart@gmail.com>
wrote:

> On Wed, Mar 05, 2025 at 09:35:27AM -0600, Nathan Bossart wrote:
> > On Wed, Mar 05, 2025 at 01:52:40PM +0100, Magnus Hagander wrote:
> >> Another option that I think would also work is to just cut down the
> details
> >> to just "The <option>--jobs</option> option allows multiple CPU cores
> to be
> >> used".
> >
> > That's fine with me.  It's probably not particularly actionable
> > information, anyway.  If anything, IMHO we should make it clear to users
> > that the parallelization is per-database (except for file transfer, which
> > is per-tablespace).  If you've just got one big database in the default
> > tablespace, --jobs won't help.
> >
> >> I think this is also slightly confusing, but maybe that's a
> >> non-native-english thing: "a good place to start is the maximum of the
> >> number of  CPU cores and tablespaces.". Am I supposed to set it to
> >> max(cpucores, ntablespaces) or to max(cpucores+ntablespaces)?
> >
> > I've always read it to mean the former.  But I'm not sure that's great
> > advice.  If you have 8 cores and 100 tablespaces, does it make sense to
> use
> > --jobs=100?  Ordinarily, I'd suggest the number of cores as the starting
> > point.
>
> Here's another attempt at the patch based on the latest discussion.
>

LGTM!

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ <http://www.hagander.net/>
 Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/>

  7. Re: doc: expand note about pg_upgrade's --jobs option

    Nathan Bossart <nathandbossart@gmail.com> — 2025-03-08T20:30:16Z

    On Sat, Mar 08, 2025 at 01:43:52AM +0100, Magnus Hagander wrote:
> LGTM!

Thanks, committed.

-- 
nathan