Re: Heads Up: cirrus-ci is shutting down June 1st

Hi,

Attached is an large incremental patch onto Bilal's version:

- I hacked enough on the pg-vm-images repo to make it store containers in
  github (on the gha_main branch, for now).  That makes the container faster
  and cheaper to retrieve.

  We might want to split the containers further (e.g. cross building and 32bit
  support), but for now I just split the docs stuff into a separate container.


- The reason that sometimes cancelling took a long time was, afaict, the use
  of always() in the compiler-warning job. That apparently prevents GHA from
  cancelling the job in a timely fashion. I turned those into !cancelled().

  Also addded a !cancelled() to SanityCheck.


- In the end I didn't like the matrix all that much, makes it considerably
  harder to understand everything and the restrictions GHA puts on it are just
  too annoying.

  I replaced it much more heavy use of yaml anchors/aliases and by updating
  the environment programattically.  I'm not sure how much better it is now.


- The macports cache keys included the run_id, I think that's a bad idea,
  because it leads to the cache being newly uploaded even if there's been no
  change. That'll lead to even more quickly churning through the cache space.


- ccache:

  - There was too much duplication around the ccache handling for my taste. I
    moved that into a yaml anchor and reused it everywhere.  By using
    ${{github.job_id}} the cache names don't need to be manually disambiguated.

  - The ccache names weren't unique enough. The run_id doesn't change during
    reruns which would lead to warnings.

  - Made it so that the cache is saved immediately after the build, so that
    cancelled builds still save the cache.


- I wanted to share commands like meson test between the tasks, made that work
  with a bit of hackery.


- I didn't see why
    find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \;
  was needed.


- I found it hard to find actual warnings in the output of CompilerWarnings,
  due to to all the configure output. I added some magic output stuff to make
  the configure output collapse once it ran.


- I converted most things to use ${{env.varname}} instead of $VARNAME or
  %VARNAME%, as that allows sharing code between windows and other OSs if one
  is a bit careful.


- Deduplicated a few other things like logging.


- Added a commit removing cirrus, mainly because I was tired of it also
  running while hacking on this.


- Also added a commit to reduce the segment size in tests to 1MB, that makes
  them a decent bit less IO intensive.


- Added a commit to just run regress/regress, makes it a lot faster to test
  "complete-ish" cycles.


- Changed sanitizer using builds to use -O2, to reduce the CPU cost a bit.

  This makes uncached builds noticeably slower, but does appear to be a
  win. But I could see counter-arguments to that too.


- Removed :detect_stack_use_after_return=0, as that's been made unnecessary
  since you "forked off" from .cirrus.tasks.yml.


A few other comments:

- All the caches in GHA apparently are branch specific, except that branches
  can access the caches of the main branch.

  I think that'll make particularly macports very expensive for cfbot.  I
  wonder if we ought to build the "base" macports cache in pg-vm-images.

  Similarly, I think we probably should do that for mingw64.


- src/tools/ci/README needs updating


- I experimented with making the slowest jobs faster by using
    meson test --slice 1/2
  that actually does nicely work. But it didn't quite seem critical for now.


Here's a run running with just regress/regress:
  https://github.com/anarazel/postgres/actions/runs/26782558353
(the CompilerWarnings ccache hit had a miss, that's why it's not faster)

Here's a (not yet completed) run with the full tests:
  https://github.com/anarazel/postgres/actions/runs/26784248887


Thoughts?


Greetings,

Andres Freund