Horton

Thread

Commits

  1. ci: macos: use cached macports install

  1. ci: Improve macos startup using a cached macports installation

    Andres Freund <andres@anarazel.de> — 2023-08-05T20:25:39Z

    Hi,

We have some issues with CI on macos and windows being too expensive (more on
that soon in a separate email). For macos most of the obviously wasted time is
spent installing packages with homebrew. Even with the package downloads being
cached, it takes about 1m20s to install them.  We can't just cache the whole
homebrew installation, because it contains a lot of pre-installed packages.

After a bunch of experimenting, I found a way to do this a lot faster: The
attached patch installs macports and installs our dependencies from
that. Because there aren't pre-existing macports packages, we can just cache
the whole thing.  Doing so naively wouldn't yield that much of a speedup,
because it takes a while to unpack a tarball (or whatnot) with as many files
as our dependencies have - that's a lot of filesystem metadata
operations. Instead the script creates a HFS+ filesystem in a file and caches
that - that's mounted within a few seconds. To further keep the size in check,
that file is compressed with zstd in the cache.

As macports has a package for IPC::Run and IO::Pty, we can use those instead
of the separate cache we had for the perl installation.

After the patch, the cached case takes ~5s to "install" and the cache is half
the size than the one for homebrew.

The comparison isn't entirely fair, because I used the occasion to not install
'make' (since meson is used for building) and llvm (we didn't enable it for
the build anyway). That gain is a bit smaller without that, but still
significant.


An alternative implementation would be to create the "base" .dmg file outside
of CI and download it onto the CI instances. But I didn't want to figure out
the hosting situation for such files, so I thought this was the best
near-medium term path.

Greetings,

Andres Freund

  2. Re: ci: Improve macos startup using a cached macports installation

    Andres Freund <andres@anarazel.de> — 2023-08-19T18:47:33Z

    Hi,

On 2023-08-05 13:25:39 -0700, Andres Freund wrote:
> We have some issues with CI on macos and windows being too expensive (more on
> that soon in a separate email). For macos most of the obviously wasted time is
> spent installing packages with homebrew. Even with the package downloads being
> cached, it takes about 1m20s to install them.  We can't just cache the whole
> homebrew installation, because it contains a lot of pre-installed packages.
> 
> After a bunch of experimenting, I found a way to do this a lot faster: The
> attached patch installs macports and installs our dependencies from
> that. Because there aren't pre-existing macports packages, we can just cache
> the whole thing.  Doing so naively wouldn't yield that much of a speedup,
> because it takes a while to unpack a tarball (or whatnot) with as many files
> as our dependencies have - that's a lot of filesystem metadata
> operations. Instead the script creates a HFS+ filesystem in a file and caches
> that - that's mounted within a few seconds. To further keep the size in check,
> that file is compressed with zstd in the cache.
> 
> As macports has a package for IPC::Run and IO::Pty, we can use those instead
> of the separate cache we had for the perl installation.
> 
> After the patch, the cached case takes ~5s to "install" and the cache is half
> the size than the one for homebrew.
> 
> The comparison isn't entirely fair, because I used the occasion to not install
> 'make' (since meson is used for building) and llvm (we didn't enable it for
> the build anyway). That gain is a bit smaller without that, but still
> significant.
> 
> 
> An alternative implementation would be to create the "base" .dmg file outside
> of CI and download it onto the CI instances. But I didn't want to figure out
> the hosting situation for such files, so I thought this was the best
> near-medium term path.

Given how significant an improvement this is in test time, and the limited
blast radius, I am planning to push this soon, unless somebody opposes that
soon.

Greetings,

Andres Freund

  3. Re: ci: Improve macos startup using a cached macports installation

    Andres Freund <andres@anarazel.de> — 2023-08-19T22:22:31Z

    On 2023-08-19 11:47:33 -0700, Andres Freund wrote:
> Given how significant an improvement this is in test time, and the limited
> blast radius, I am planning to push this soon, unless somebody opposes that
> soon.

And done.