Re: ci: Improve macos startup using a cached macports installation

Hi,

On 2023-08-05 13:25:39 -0700, Andres Freund wrote:
> We have some issues with CI on macos and windows being too expensive (more on
> that soon in a separate email). For macos most of the obviously wasted time is
> spent installing packages with homebrew. Even with the package downloads being
> cached, it takes about 1m20s to install them.  We can't just cache the whole
> homebrew installation, because it contains a lot of pre-installed packages.
> 
> After a bunch of experimenting, I found a way to do this a lot faster: The
> attached patch installs macports and installs our dependencies from
> that. Because there aren't pre-existing macports packages, we can just cache
> the whole thing.  Doing so naively wouldn't yield that much of a speedup,
> because it takes a while to unpack a tarball (or whatnot) with as many files
> as our dependencies have - that's a lot of filesystem metadata
> operations. Instead the script creates a HFS+ filesystem in a file and caches
> that - that's mounted within a few seconds. To further keep the size in check,
> that file is compressed with zstd in the cache.
> 
> As macports has a package for IPC::Run and IO::Pty, we can use those instead
> of the separate cache we had for the perl installation.
> 
> After the patch, the cached case takes ~5s to "install" and the cache is half
> the size than the one for homebrew.
> 
> The comparison isn't entirely fair, because I used the occasion to not install
> 'make' (since meson is used for building) and llvm (we didn't enable it for
> the build anyway). That gain is a bit smaller without that, but still
> significant.
> 
> 
> An alternative implementation would be to create the "base" .dmg file outside
> of CI and download it onto the CI instances. But I didn't want to figure out
> the hosting situation for such files, so I thought this was the best
> near-medium term path.

Given how significant an improvement this is in test time, and the limited
blast radius, I am planning to push this soon, unless somebody opposes that
soon.

Greetings,

Andres Freund