Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

ci: Add GitHub Actions based CI
- 9c126063b19a 19 (unreleased) landed
ci: Remove support for cirrus-ci based CI
- 68c8a365d4dc 19 (unreleased) landed

Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-04-09T20:55:14Z

Hi,

As the subject says, cirrus-ci, which cfbot uses to run CI and that one can
(for now) enable on one's own repository, is shutting down.

https://cirruslabs.org/ burries the lede a bit, but it has further down:
  "Cirrus CI will shut down effective Monday, June 1, 2026."

I can't say I'm terribly surprised, they had been moving a lot slower in the
last few years.

The shutdown window is pretty short, so we'll have to do something soon. Glad
that it didn't happen a few months ago, putting the shutdown before the
feature freeze. This is probably close to the least bad time it could happen
with a short window.


I think having cfbot and CI that one could run on ones own repository, without
sending a mail to the community, has improved the development process a lot.
So clearly we're going to have to do something.  I certainly could not have
done stuff like AIO without it.


I'd be interested in feedback about how high folks value different aspects:

1) CI software can be self hosted

   E.g. to prevent at least the cfbot case from being unpredictably abandoned
   again.


2) CI software is open source

   E.g. out of a principled stance, or control concerns.


3) CI runs quickly

   This matters e.g. for accepting running in containers and whether it's
   crucial to be able to have our images with everything pre-installed.


4) CI tests as many operating systems as possible

   A lot of system just support linux, plenty support macos, some support
   windows. Barely any support anything beyond that.


5) CI can be enabled on one's own repositories

   Cfbot obviously allows everyone to test patches some way, but sending patch
   sets to the list just to get a CI run obviously gets noisy quite fast.

   There are plenty of open source CI solutions, but clearly it's not viable
   for everyone to set that up for themselves. Plenty providers do allow doing
   so, but the overlap of this, open source (2), multiple platforms (4) is
   small if it exists.


6) There need to be free credits for running at least some CI on one's own
   repository

   This makes the overlapping constraints mentioned in 5) even smaller.

   There are several platforms that do provide a decent amount of CI for a
   monthly charge of < 10 USD.


7) Provide CI compute for "well known contributors" for free in their own
   repositories

   An alternative to 6) - with some CI solutions - can be to add folks to some
   team that allows them to use community resources (which so far have been
   donated).  The problem with that is that it's administratively annoying,
   because one does need to be careful, or CI will be used to do
   cryptocurrency mining or such within a few days.


For some context about how much CI we have been running, here's the daily
average for cfbot and postgres/postgres CI:

- 1464 core hours (full cores, not SMT), all CI jobs use 4 cores

- 396 core hours of which were windows (visible due to the licensing cost)

- 40 GB of artifacts

- 83 GB of artifacts downloaded externally

- doesn't include macos, which I can't track as easily, due to being self
  hosted runners, rather than running on GCP, which provided the above numbers


Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Thomas Munro <thomas.munro@gmail.com> — 2026-04-09T23:29:45Z

On Fri, Apr 10, 2026 at 8:55 AM Andres Freund <andres@anarazel.de> wrote:
> 4) CI tests as many operating systems as possible
>
>    A lot of system just support linux, plenty support macos, some support
>    windows. Barely any support anything beyond that.

Nested virtualisation to the rescue?

https://github.com/cross-platform-actions/action

Re: Heads Up: cirrus-ci is shutting down June 1st

Jelte Fennema <postgres@jeltef.nl> — 2026-04-10T11:31:38Z

On Thu, 9 Apr 2026 at 22:55, Andres Freund <andres@anarazel.de> wrote:
> I'd be interested in feedback about how high folks value different aspects:

My thoughts below.

> 1) CI software can be self hosted
>
>    E.g. to prevent at least the cfbot case from being unpredictably abandoned
>    again.

Low, I personally don't want to manage self hosting it. Self-hosted
software can just as easily be abandoned. i.e. I'd expect GitHub
Actions to outlive underfunded open source CI software.

> 2) CI software is open source
>
>    E.g. out of a principled stance, or control concerns.

I don't care

> 3) CI runs quickly
>
>    This matters e.g. for accepting running in containers and whether it's
>    crucial to be able to have our images with everything pre-installed.

Important. We should definitely be able to pre-install stuff for most
OSes. I think running stuff in containers would be fine.

> 4) CI tests as many operating systems as possible

I think we at minimum need linux+macos+windows. Windows is by far the
system that fails most often for me. BSDs would be good, but they
often tend to be fine if osx and linux work. Personally for me I think
on Cirrus The BSDs didn't meet the useful signal to flakiness noise
ratio (i.e. they tended to mostly break randomly for me).

> 5) CI can be enabled on one's own repositories
> ...
> 6) There need to be free credits for running at least some CI on one's own
>    repository
> ...
> 7) Provide CI compute for "well known contributors" for free in their own
>    repositories

I would say it's a hard requirement that people can run CI without
spamming the list. I don't think that necessarily has to be in
someone's own repository. e.g. having a way for committers to give
e.g. some limited number (e.g. 100) of CI hours to someone submitting
their first patch seems fairly low risk. If they continue contributing
they can receive a recurring number or unlimited access.

On Thu, 9 Apr 2026 at 22:55, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> As the subject says, cirrus-ci, which cfbot uses to run CI and that one can
> (for now) enable on one's own repository, is shutting down.
>
> https://cirruslabs.org/ burries the lede a bit, but it has further down:
>   "Cirrus CI will shut down effective Monday, June 1, 2026."
>
> I can't say I'm terribly surprised, they had been moving a lot slower in the
> last few years.
>
> The shutdown window is pretty short, so we'll have to do something soon. Glad
> that it didn't happen a few months ago, putting the shutdown before the
> feature freeze. This is probably close to the least bad time it could happen
> with a short window.
>
>
> I think having cfbot and CI that one could run on ones own repository, without
> sending a mail to the community, has improved the development process a lot.
> So clearly we're going to have to do something.  I certainly could not have
> done stuff like AIO without it.
>
>
> I'd be interested in feedback about how high folks value different aspects:
>
> 1) CI software can be self hosted
>
>    E.g. to prevent at least the cfbot case from being unpredictably abandoned
>    again.
>
>
> 2) CI software is open source
>
>    E.g. out of a principled stance, or control concerns.
>
>
> 3) CI runs quickly
>
>    This matters e.g. for accepting running in containers and whether it's
>    crucial to be able to have our images with everything pre-installed.
>
>
> 4) CI tests as many operating systems as possible
>
>    A lot of system just support linux, plenty support macos, some support
>    windows. Barely any support anything beyond that.
>
>
> 5) CI can be enabled on one's own repositories
>
>    Cfbot obviously allows everyone to test patches some way, but sending patch
>    sets to the list just to get a CI run obviously gets noisy quite fast.
>
>    There are plenty of open source CI solutions, but clearly it's not viable
>    for everyone to set that up for themselves. Plenty providers do allow doing
>    so, but the overlap of this, open source (2), multiple platforms (4) is
>    small if it exists.
>
>
> 6) There need to be free credits for running at least some CI on one's own
>    repository
>
>    This makes the overlapping constraints mentioned in 5) even smaller.
>
>    There are several platforms that do provide a decent amount of CI for a
>    monthly charge of < 10 USD.
>
>
> 7) Provide CI compute for "well known contributors" for free in their own
>    repositories
>
>    An alternative to 6) - with some CI solutions - can be to add folks to some
>    team that allows them to use community resources (which so far have been
>    donated).  The problem with that is that it's administratively annoying,
>    because one does need to be careful, or CI will be used to do
>    cryptocurrency mining or such within a few days.
>
>
> For some context about how much CI we have been running, here's the daily
> average for cfbot and postgres/postgres CI:
>
> - 1464 core hours (full cores, not SMT), all CI jobs use 4 cores
>
> - 396 core hours of which were windows (visible due to the licensing cost)
>
> - 40 GB of artifacts
>
> - 83 GB of artifacts downloaded externally
>
> - doesn't include macos, which I can't track as easily, due to being self
>   hosted runners, rather than running on GCP, which provided the above numbers
>
>
> Greetings,
>
> Andres Freund
>
>

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-04-10T11:51:24Z

Hi,

On Fri, 10 Apr 2026 at 14:31, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
>
> On Thu, 9 Apr 2026 at 22:55, Andres Freund <andres@anarazel.de> wrote:
> > I'd be interested in feedback about how high folks value different aspects:
>
> My thoughts below.

I agree with all of Jelte's points except:

> > 4) CI tests as many operating systems as possible
>
> I think we at minimum need linux+macos+windows. Windows is by far the
> system that fails most often for me. BSDs would be good, but they
> often tend to be fine if osx and linux work. Personally for me I think
> on Cirrus The BSDs didn't meet the useful signal to flakiness noise
> ratio (i.e. they tended to mostly break randomly for me).

I think BSDs are quite capable of catching issues that others can't
catch. That has at least been my experience with OpenBSD.

However, I agree that OpenBSD and NetBSD tasks are flaky; I think that
is mostly because we generate these VM images from scratch (i.e. other
operating systems' VM images were already available on GCP). I don't
think FreeBSD is flaky.

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Alexander Korotkov <aekorotkov@gmail.com> — 2026-04-10T12:23:16Z

Hi!

On Thu, Apr 9, 2026 at 11:55 PM Andres Freund <andres@anarazel.de> wrote:
>
> As the subject says, cirrus-ci, which cfbot uses to run CI and that one can
> (for now) enable on one's own repository, is shutting down.
>
> https://cirruslabs.org/ burries the lede a bit, but it has further down:
>   "Cirrus CI will shut down effective Monday, June 1, 2026."
>
> I can't say I'm terribly surprised, they had been moving a lot slower in the
> last few years.
>
> The shutdown window is pretty short, so we'll have to do something soon. Glad
> that it didn't happen a few months ago, putting the shutdown before the
> feature freeze. This is probably close to the least bad time it could happen
> with a short window.


+1

>
> I think having cfbot and CI that one could run on ones own repository, without
> sending a mail to the community, has improved the development process a lot.
> So clearly we're going to have to do something.  I certainly could not have
> done stuff like AIO without it.
>
>
> I'd be interested in feedback about how high folks value different aspects:
>
> 1) CI software can be self hosted
>
>    E.g. to prevent at least the cfbot case from being unpredictably abandoned
>    again.
>
>
> 2) CI software is open source
>
>    E.g. out of a principled stance, or control concerns.
>
>
> 3) CI runs quickly
>
>    This matters e.g. for accepting running in containers and whether it's
>    crucial to be able to have our images with everything pre-installed.
>
>
> 4) CI tests as many operating systems as possible
>
>    A lot of system just support linux, plenty support macos, some support
>    windows. Barely any support anything beyond that.
>
>
> 5) CI can be enabled on one's own repositories
>
>    Cfbot obviously allows everyone to test patches some way, but sending patch
>    sets to the list just to get a CI run obviously gets noisy quite fast.
>
>    There are plenty of open source CI solutions, but clearly it's not viable
>    for everyone to set that up for themselves. Plenty providers do allow doing
>    so, but the overlap of this, open source (2), multiple platforms (4) is
>    small if it exists.
>
>
> 6) There need to be free credits for running at least some CI on one's own
>    repository
>
>    This makes the overlapping constraints mentioned in 5) even smaller.
>
>    There are several platforms that do provide a decent amount of CI for a
>    monthly charge of < 10 USD.
>
>
> 7) Provide CI compute for "well known contributors" for free in their own
>    repositories
>
>    An alternative to 6) - with some CI solutions - can be to add folks to some
>    team that allows them to use community resources (which so far have been
>    donated).  The problem with that is that it's administratively annoying,
>    because one does need to be careful, or CI will be used to do
>    cryptocurrency mining or such within a few days.

It's hard for me to judge priorities, but I have a proposal on how we
can try to handle this.

Migrate to Open Source CI software, and run it on (cheap) cloud + get
sponsorship to cover the migration cost.  This should protect us from
disasters like this.  In worst case we would need to loop for
different cloud or different sponsor.

Provide CI workflow for GIthub Actions on our repository.  This
wouldn't provide the plurality of platforms that we have now, but at
least everybody can get some basic CI coverage for free.

What do you think?

------
Regards,
Alexander Korotkov
Supabase

Re: Heads Up: cirrus-ci is shutting down June 1st

Alexander Korotkov <aekorotkov@gmail.com> — 2026-04-10T12:24:10Z

On Fri, Apr 10, 2026 at 3:23 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> On Thu, Apr 9, 2026 at 11:55 PM Andres Freund <andres@anarazel.de> wrote:
> >
> > As the subject says, cirrus-ci, which cfbot uses to run CI and that one can
> > (for now) enable on one's own repository, is shutting down.
> >
> > https://cirruslabs.org/ burries the lede a bit, but it has further down:
> >   "Cirrus CI will shut down effective Monday, June 1, 2026."
> >
> > I can't say I'm terribly surprised, they had been moving a lot slower in the
> > last few years.
> >
> > The shutdown window is pretty short, so we'll have to do something soon. Glad
> > that it didn't happen a few months ago, putting the shutdown before the
> > feature freeze. This is probably close to the least bad time it could happen
> > with a short window.
>
>
> +1
>
> >
> > I think having cfbot and CI that one could run on ones own repository, without
> > sending a mail to the community, has improved the development process a lot.
> > So clearly we're going to have to do something.  I certainly could not have
> > done stuff like AIO without it.
> >
> >
> > I'd be interested in feedback about how high folks value different aspects:
> >
> > 1) CI software can be self hosted
> >
> >    E.g. to prevent at least the cfbot case from being unpredictably abandoned
> >    again.
> >
> >
> > 2) CI software is open source
> >
> >    E.g. out of a principled stance, or control concerns.
> >
> >
> > 3) CI runs quickly
> >
> >    This matters e.g. for accepting running in containers and whether it's
> >    crucial to be able to have our images with everything pre-installed.
> >
> >
> > 4) CI tests as many operating systems as possible
> >
> >    A lot of system just support linux, plenty support macos, some support
> >    windows. Barely any support anything beyond that.
> >
> >
> > 5) CI can be enabled on one's own repositories
> >
> >    Cfbot obviously allows everyone to test patches some way, but sending patch
> >    sets to the list just to get a CI run obviously gets noisy quite fast.
> >
> >    There are plenty of open source CI solutions, but clearly it's not viable
> >    for everyone to set that up for themselves. Plenty providers do allow doing
> >    so, but the overlap of this, open source (2), multiple platforms (4) is
> >    small if it exists.
> >
> >
> > 6) There need to be free credits for running at least some CI on one's own
> >    repository
> >
> >    This makes the overlapping constraints mentioned in 5) even smaller.
> >
> >    There are several platforms that do provide a decent amount of CI for a
> >    monthly charge of < 10 USD.
> >
> >
> > 7) Provide CI compute for "well known contributors" for free in their own
> >    repositories
> >
> >    An alternative to 6) - with some CI solutions - can be to add folks to some
> >    team that allows them to use community resources (which so far have been
> >    donated).  The problem with that is that it's administratively annoying,
> >    because one does need to be careful, or CI will be used to do
> >    cryptocurrency mining or such within a few days.
>
> It's hard for me to judge priorities, but I have a proposal on how we
> can try to handle this.
>
> Migrate to Open Source CI software, and run it on (cheap) cloud + get
> sponsorship to cover the migration cost.

Sorry, I meant sponsorship to cover the cloud cost.

------
Regards,
Alexander Korotkov
Supabase

Re: Heads Up: cirrus-ci is shutting down June 1st

Peter Eisentraut <peter@eisentraut.org> — 2026-04-10T13:05:27Z

On 09.04.26 22:55, Andres Freund wrote:
> I'd be interested in feedback about how high folks value different aspects:
> 
> 1) CI software can be self hosted
> 
>     E.g. to prevent at least the cfbot case from being unpredictably abandoned
>     again.
> 
> 
> 2) CI software is open source
> 
>     E.g. out of a principled stance, or control concerns.

I think we should work toward that in the long run.  Open-source 
software should also have an open-source (and distributed, and 
privacy-respecting, and reusable, etc.) development process.

In the short run, meaning something that is plausible to get ready 
between now and June/July, using some stopgap from an existing 
established provider (such as GH actions) would probably be better.

> 3) CI runs quickly
> 
>     This matters e.g. for accepting running in containers and whether it's
>     crucial to be able to have our images with everything pre-installed.
> 
> 
> 4) CI tests as many operating systems as possible
> 
>     A lot of system just support linux, plenty support macos, some support
>     windows. Barely any support anything beyond that.
> 
> 
> 5) CI can be enabled on one's own repositories
> 
>     Cfbot obviously allows everyone to test patches some way, but sending patch
>     sets to the list just to get a CI run obviously gets noisy quite fast.
> 
>     There are plenty of open source CI solutions, but clearly it's not viable
>     for everyone to set that up for themselves. Plenty providers do allow doing
>     so, but the overlap of this, open source (2), multiple platforms (4) is
>     small if it exists.

This is the most important one, for me.

I think it would be even more useful if one could run the whole thing, 
or most of the thing, locally.  I mean, I can run all kinds of VMs 
locally, all the pieces of this already exist.  But it needs some 
integration to build the images locally, and then run the build and test 
processes in this images.  This wouldn't cover everything (e.g., can't 
virtualize macOS unless on macOS, IIRC), but I shouldn't really need to 
push my code half-way around the world just to do a build run on NetBSD. 
  This could be someone's $season of code project.

> 7) Provide CI compute for "well known contributors" for free in their own
>     repositories
> 
>     An alternative to 6) - with some CI solutions - can be to add folks to some
>     team that allows them to use community resources (which so far have been
>     donated).  The problem with that is that it's administratively annoying,
>     because one does need to be careful, or CI will be used to do
>     cryptocurrency mining or such within a few days.

In a way, well known contributors can fend for themselves.  We want to 
get as many new or occasional contributors to run this so that the 
patches build and test successfully before anyone else has to look at them.

Re: Heads Up: cirrus-ci is shutting down June 1st

Bruce Momjian <bruce@momjian.us> — 2026-04-10T13:27:47Z

On Fri, Apr 10, 2026 at 01:31:38PM +0200, Jelte Fennema-Nio wrote:
> On Thu, 9 Apr 2026 at 22:55, Andres Freund <andres@anarazel.de> wrote:
> > I'd be interested in feedback about how high folks value different aspects:
> 
> My thoughts below.
> 
> > 1) CI software can be self hosted
> >
> >    E.g. to prevent at least the cfbot case from being unpredictably abandoned
> >    again.
> 
> Low, I personally don't want to manage self hosting it. Self-hosted
> software can just as easily be abandoned. i.e. I'd expect GitHub
> Actions to outlive underfunded open source CI software.

Uh, I actually think the opposite.  while proprietary software doesn't
disappear, it seems to become obsolete (underfunded development) or
prohibitively expensive sooner than open source.  The dataase industry
has certainly shown that in the past 30 years.  Also, four months ago
Github wanted to charge for self-hosted actions, which supports
"prohibitively expensive":

	https://www.reddit.com/r/devops/comments/1po8hj5/github_actions_introducing_a_perminute_fee_for/

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

Re: Heads Up: cirrus-ci is shutting down June 1st

Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-13T08:34:44Z

On 09/04/2026 23:55, Andres Freund wrote:
> As the subject says, cirrus-ci, which cfbot uses to run CI and that one can
> (for now) enable on one's own repository, is shutting down.
> 
> https://cirruslabs.org/ burries the lede a bit, but it has further down:
>    "Cirrus CI will shut down effective Monday, June 1, 2026."
> 
> I can't say I'm terribly surprised, they had been moving a lot slower in the
> last few years.

Darn, I liked Cirrus CI. One reason being precisely that it has been 
stable, i.e. moved slowly, for years :-).

> I think having cfbot and CI that one could run on ones own repository, without
> sending a mail to the community, has improved the development process a lot.
> So clearly we're going to have to do something.  I certainly could not have
> done stuff like AIO without it.

+1. I rely heavily on cirrus CI nowadays to validate before I push.

> I'd be interested in feedback about how high folks value different aspects:
> 
> 1) CI software can be self hosted
> 
>     E.g. to prevent at least the cfbot case from being unpredictably abandoned
>     again.
> 
> 
> 2) CI software is open source
> 
>     E.g. out of a principled stance, or control concerns.

These probably go together.

I think it's important that you can self-host. Even with cirrus-ci I 
actually wished there was an easy way to run the jobs locally. I don't 
know how often I'd really do it, but especially developing and testing 
the ci yaml files is painful when you can't run it locally.

> 3) CI runs quickly
> 
>     This matters e.g. for accepting running in containers and whether it's
>     crucial to be able to have our images with everything pre-installed.

Pretty important. "quickly" is pretty subjective though, I'm not sure 
what number to put to it. Cirrus-CI has felt fast enough.

> 4) CI tests as many operating systems as possible
> 
>     A lot of system just support linux, plenty support macos, some support
>     windows. Barely any support anything beyond that.

Windows support is pretty important as it's different enough from 
others. Macos is definitely good to have too. For others, we have the 
buildfarm.

> 5) CI can be enabled on one's own repositories
> 
>     Cfbot obviously allows everyone to test patches some way, but sending patch
>     sets to the list just to get a CI run obviously gets noisy quite fast.
> 
>     There are plenty of open source CI solutions, but clearly it's not viable
>     for everyone to set that up for themselves. Plenty providers do allow doing
>     so, but the overlap of this, open source (2), multiple platforms (4) is
>     small if it exists.

This is important. I run the CI as part of development on my own 
branches all the time.

If it's easy to self-host, that might cover it.

> 6) There need to be free credits for running at least some CI on one's own
>     repository
> 
>     This makes the overlapping constraints mentioned in 5) even smaller.
> 
>     There are several platforms that do provide a decent amount of CI for a
>     monthly charge of < 10 USD.

Not important. For running on one's own repository, it's totally 
reasonable that you pay for it yourself. Especially if you can self-host 
for free.

> 7) Provide CI compute for "well known contributors" for free in their own
>     repositories
> 
>     An alternative to 6) - with some CI solutions - can be to add folks to some
>     team that allows them to use community resources (which so far have been
>     donated).  The problem with that is that it's administratively annoying,
>     because one does need to be careful, or CI will be used to do
>     cryptocurrency mining or such within a few days.

Not important. Active contributors can easily pay for what they use, or 
self-host.

- Heikki

Re: Heads Up: cirrus-ci is shutting down June 1st

David Steele <david@pgbackrest.org> — 2026-04-13T11:53:39Z

On 4/10/26 06:29, Thomas Munro wrote:
> On Fri, Apr 10, 2026 at 8:55 AM Andres Freund <andres@anarazel.de> wrote:
>> 4) CI tests as many operating systems as possible
>>
>>     A lot of system just support linux, plenty support macos, some support
>>     windows. Barely any support anything beyond that.
> 
> Nested virtualisation to the rescue?
> 
> https://github.com/cross-platform-actions/action

I used this to migrate our FreeBSD tests [1] and it worked out OK. The 
only downside is it doesn't look like you can split out steps so all the 
commands end up logged together.

Regards,
-David

[1] 
https://github.com/pgbackrest/pgbackrest/blob/main/.github/workflows/test.yml#L148

Re: Heads Up: cirrus-ci is shutting down June 1st

David E. Wheeler <david@justatheory.com> — 2026-04-13T14:34:15Z

Hi,

I’ve started thinking about moving away from GitHub actions myself, and was wondering what else was out there that fulfills a bunch of these needs. Feedback I got and some brief research turned up Woodpecker CI[0]

[0]: https://woodpecker-ci.org/

On Apr 13, 2026, at 04:34, Heikki Linnakangas <hlinnaka@iki.fi> wrote:

> These probably go together.
> 
> I think it's important that you can self-host. Even with cirrus-ci I actually wished there was an easy way to run the jobs locally. I don't know how often I'd really do it, but especially developing and testing the ci yaml files is painful when you can't run it locally.

While Woodpecker promotes its Docker images, esp. for integration with Codeberg and other Forgejo services, it’s a Go app so compiles for quite a lot of platforms, and has a “local mode” in which, from what I understand, you can run it on whatever trusted hardware you’d like.

So if we have, say, a Mac Mini plus an arm and amd system capable of virtualizing Linux, BSD, etc., perhaps we’d be able to get the coverage we need and host the results in a self-hosted Woodpecker service?

As I say, I’ve just started to kind of cast about for alternatives, so don’t know a lot about it myself, but on the surface it looks promising.

Best,

David

Re: Heads Up: cirrus-ci is shutting down June 1st

Thomas Munro <thomas.munro@gmail.com> — 2026-04-14T00:57:27Z

On Mon, Apr 13, 2026 at 11:53 PM David Steele <david@pgbackrest.org> wrote:
> On 4/10/26 06:29, Thomas Munro wrote:
> > Nested virtualisation to the rescue?
> >
> > https://github.com/cross-platform-actions/action
>
> I used this to migrate our FreeBSD tests [1] and it worked out OK. The
> only downside is it doesn't look like you can split out steps so all the
> commands end up logged together.

- name: Run Test
uses: cross-platform-actions/action@v1.0.0
with:
operating_system: freebsd
version: ${{matrix.os.version}}
run: |
uname -a
sudo pkg update && sudo pkg upgrade -y libiconv && sudo
pkg install -y bash git postgresql-libpqxx pkgconf libxml2 gmake perl5
libyaml p5-YAML-LibYAML rsync meson
cd .. && perl ${GITHUB_WORKSPACE?}/test/test.pl --vm-max=2
--no-coverage --no-valgrind --module=command --test=backup
--test=info --test=archive-push

Nice!

I guess the problems with this are:

1. It has to install the packages every time because it's not yet
using a pre-prepared image.
2. It has no ccache memory.
3. It has lost all that user-friendly stuff like artefact
archival/browsing, core file debugging etc.
4. IIRC the log URLs are not "public", you have to have to be logged
into an account to view them.

(That 4th point was one of Cirrus's unique advantages at the time we
selected it. We wanted to be able to share URLs for discussion on the
mailing list without requiring everyone to be a GitHub user.)

Perhaps for point 1, we could publish fast-start qemu images for
Debian, FreeBSD, NetBSD, OpenBSD*. The pg-vm-images repo that Andres
and Bilal maintain currently uploads images to Google Cloud's image
repository where Cirrus VMs can boot from them, but it could instead
publish qemu images to our own public URLs. I'm picturing a bunch of
images available as
https://ci.postgresql.org/images/qemu/arm64/{freebsd-15,debian-13,...}-with-postgresql-dependencies.img.
The point of per-arch variants would be to match common hosts for fast
kernel hypervisor support, eg on a Mac you want arm64, though you
could still run amd64 slowly if you need to. Could even do ppc and
riscv with emulation.

Qemu images should hopefully be usable in many different environments:

1. We could run them locally with some one-button command, and also
have images you can log into and hack on if you want.
2. We could run all of them or just the license-encumbered ones on
public clouds (not through a CI service) with some one-button command,
if you have an account.
3. We could use them in people's private GitHub/GitLab/... accounts
as you showed, just add
image_url=https://ci.postgresql.org/images/qemu/....
4. Cfbot could do any of those things, not sure what would be best.

For the license-encumbered OSes, we could at least make disk images or
archives containing a MacPorts installation or
bunch-of-installed-libraries-for-Windows, but not including the OS.
Just mount/unpack as /opt or C:\pg-packages or whatever, I guess, if
you can figure out how to get a VM running ... somewhere. Perhaps
there is some way to make project-owned resources (MacMinis, Windows
VMs) available to our community too, but IDK how that would work.

Some random half-baked thoughts about the ccache, browsing, etc problems:

1. Local qemu: we could use overlay images so that your downloaded
copy of X-with-postgres-dependencies.img remains read-only. Create a
new empty overlay image for each clean run, and if you need to inspect
logs, core files, you can just log in before the next run wipes it.
2. Local qemu: we could mount a separate disk image as /cache that
survives between runs and can be wiped any time.
3. Public CI system like GitHub actions: I suppose we could run our
own ccache, artefact, log hosting service that it could push to...
that was something I already wondered about under Cirrus due to
various disk space and retention problems... but I'm quite hesitant to
get tangled up in running "public" services and unsure how you'd
control access.

I would at least like to think about trying to make cfbot
capitalism-proof. I may be underestimating the difficulty, but I keep
wondering if cfbot should at least be able to do everything itself,
with some combination of local qemu, qemu-on-project-Mac-fleet, and
public cloud VMs controlled directly. It doesn't really *need* to
depend on ephemeral venture capital-powered CI companies, it was just
nice to make it use the exact same CI setup as you could use for
yourself in your GitHub account. I'm imagining that it would still
push branches to GitHub, since that's a nice interface to browse code
on, and I suppose it might even be possible to publish our own
minimalist GitHub plugin that allows cfbot to push its green/red
result indicators to it since that's clearly something that external
providers can do (as well as pushing them to the commitfest UI as
now). But if you clicked them, you'd be taken to a really primitive
cfbot web interface where you could browse logs and artefacts retained
for N days. In other words, an extremely cut down and limited
"let's-make-our-own-CI" project, which doesn't have to tackle the much
harder "let's-make-our-own-semi-public-CI-platform" project. I like
the idea of at least having such a mode as an insurance policy anyway,
but I'm not sure what nitty gritty details might make it hard to pull
off... In this thought experiment, people could continue to work
separately on making personal CI work in various ways, GitHub, GitLab,
whatever else, and local, and all ways of doing it would be using the
same scripts and VM images.

* ... and AFAIK we could add illumos to the set if we wanted, in the
past a couple of us tried to get that going but ran into ... I think
it was driver problems? ... when using GCP VMs, but it definitely
works in qemu VMs as that cross-platform-actions project shows. Every
OS project makes sure it can boot in qemu. Even AIX can boot in qemu,
if you have a license.

Re: Heads Up: cirrus-ci is shutting down June 1st

Robert Haas <robertmhaas@gmail.com> — 2026-04-17T18:50:53Z

On Thu, Apr 9, 2026 at 4:55 PM Andres Freund <andres@anarazel.de> wrote:
> I'd be interested in feedback about how high folks value different aspects:
>
> 1) CI software can be self hosted
> 2) CI software is open source
> 3) CI runs quickly
> 4) CI tests as many operating systems as possible
> 5) CI can be enabled on one's own repositories
> 6) There need to be free credits for running at least some CI on one's own
>    repository
> 7) Provide CI compute for "well known contributors" for free in their own
>    repositories

I think we need most of these things. CI has become an indispensable
development tool for most of us at this point. If (1) and (2) then (6)
and (7) are less necessary, and conversely. (5) seems pretty critical;
as long as I took to get CI set up on my own repo, I now use it
extensively. (4) is less critical: we could probably live with just
Linux and Windows in a pinch; adding MacOS and/or *BSD would be nicer.

> For some context about how much CI we have been running, here's the daily
> average for cfbot and postgres/postgres CI:
>
> - 1464 core hours (full cores, not SMT), all CI jobs use 4 cores
> - 396 core hours of which were windows (visible due to the licensing cost)
> - 40 GB of artifacts
> - 83 GB of artifacts downloaded externally
> - doesn't include macos, which I can't track as easily, due to being self
>   hosted runners, rather than running on GCP, which provided the above numbers

I wonder if we should be looking to add more heuristics to the system
to try to reduce these numbers. For example, just browsing through the
cfbot queue, I found this:

heapam_tuple_complete_speculative : remove unnecessary tuple fetch

https://cirrus-ci.com/github/postgresql-cfbot/postgresql/cf%2F6613

This patch removes six lines of code and adds none. There are four
messages on the thread. We've done 14 complete CI runs. That might be
an extreme example, but I just don't know if repeatedly running CI on
small patches that aren't being actively updated is really what we
want to be doing.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: Heads Up: cirrus-ci is shutting down June 1st

Michael Paquier <michael@paquier.xyz> — 2026-04-17T21:41:42Z

On Fri, Apr 17, 2026 at 02:50:53PM -0400, Robert Haas wrote:
> This patch removes six lines of code and adds none. There are four
> messages on the thread. We've done 14 complete CI runs. That might be
> an extreme example, but I just don't know if repeatedly running CI on
> small patches that aren't being actively updated is really what we
> want to be doing.

Yes, starting with a low threshold should have little impact.  I
suspect that we could take it slow, say by testing much less patches
that have a max of N lines touched (20~50?), and shave in resource
usage.  This would not change much how useful the information provided
is.
--
Michael

Re: Heads Up: cirrus-ci is shutting down June 1st

Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-17T21:48:04Z

Michael Paquier <michael@paquier.xyz> writes:
> On Fri, Apr 17, 2026 at 02:50:53PM -0400, Robert Haas wrote:
>> This patch removes six lines of code and adds none. There are four
>> messages on the thread. We've done 14 complete CI runs. That might be
>> an extreme example, but I just don't know if repeatedly running CI on
>> small patches that aren't being actively updated is really what we
>> want to be doing.

> Yes, starting with a low threshold should have little impact.  I
> suspect that we could take it slow, say by testing much less patches
> that have a max of N lines touched (20~50?), and shave in resource
> usage.  This would not change much how useful the information provided
> is.

I think running a test promptly after a new patch submission is
useful, even for small patches.  I agree that the periodic re-tests
for bit-rot could be scaled back a lot.

			regards, tom lane

Re: Heads Up: cirrus-ci is shutting down June 1st

Euler Taveira <euler@eulerto.com> — 2026-04-18T02:19:24Z

On Fri, Apr 17, 2026, at 6:48 PM, Tom Lane wrote:
>
> I think running a test promptly after a new patch submission is
> useful, even for small patches.  I agree that the periodic re-tests
> for bit-rot could be scaled back a lot.
>

That's my opinion too. This is particularly important for first-time
contributors that may not know about the Postgres development process. For
regular contributors, I expect that they have a CI setup and submit a new
version only after the patch passes CI in its own repository.

I'm not sure about restricting the CI runs to small patches. Although it is a
minority, there are small patches that has a big potential to break things.
Maybe an alternative to small and/or high-frequency patches is to not run them
automatically but have a mechanism to trigger them manually once detected. The
author or even one of the reviewers can trigger it.

-- 
Euler Taveira
EDB   https://www.enterprisedb.com/

Re: Heads Up: cirrus-ci is shutting down June 1st

Jelte Fennema <postgres@jeltef.nl> — 2026-05-18T21:22:12Z

On Thu, 9 Apr 2026 at 22:55, Andres Freund <andres@anarazel.de> wrote:
> https://cirruslabs.org/ burries the lede a bit, but it has further down:
>   "Cirrus CI will shut down effective Monday, June 1, 2026."

June 1st is getting really close. *In less than two weeks we won't have
a working CI anymore*. Effectively that's probably one week of calendar
dev time that's left, because almost everyone is at PGConf.dev right
now.

So today I decided to take a stab at an initial GitHub Actions yaml
file, as that seemed like the only viable option within that timeline.
Note: Claude Code wrote this file entirely, following extensive
back-and-forth with me after repeatadly getting red builds due to some
differences between Cirrus and Github Actions.

But finally, I managed to get a green build for all systems that we
support on Cirrus CI!

IMPORTANT CONTEXT: I only did a cursory review of the workflow file that
Claude created. So there's probably still a bunch of cleanup to do, but
I at least wanted to share this initial base. I don't know if I'll have
any more time to work on this before June 1st. I have a newborn that's
taking up a lot of my spare time. So, I'd be very happy if someone else
wants to take this patch over and get it to a committable state.

A few things (apart from more extensive review) that I think should be
improved soon (but maybe not before the first commit):
1. io_uring support is disabled. I couldn't get it to work on the GitHub
   Actions runners, I think it's disabled in the host kernel. @Andres
2. It's not using pre-built images at the moment, except for the Linux
   docker images I think. So it re-downloads a bunch of dependencies for
   every build for most OSes. @Bilal or @Andres
3. There's currently no integration with the CFBot or commitfest yet. @Thomas

P.S. This is not an attempt to decide on the proprietary vs self-hosted
opensource discussion. Self hosting might still be the best long-term
solution. I just don't realistically see that happening within two
weeks.

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-05-18T22:27:52Z

Hi,

On Tue, 19 May 2026 at 00:22, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
>
> On Thu, 9 Apr 2026 at 22:55, Andres Freund <andres@anarazel.de> wrote:
> > https://cirruslabs.org/ burries the lede a bit, but it has further down:
> >   "Cirrus CI will shut down effective Monday, June 1, 2026."
>
> June 1st is getting really close. *In less than two weeks we won't have
> a working CI anymore*. Effectively that's probably one week of calendar
> dev time that's left, because almost everyone is at PGConf.dev right
> now.
>
> So today I decided to take a stab at an initial GitHub Actions yaml
> file, as that seemed like the only viable option within that timeline.
> Note: Claude Code wrote this file entirely, following extensive
> back-and-forth with me after repeatadly getting red builds due to some
> differences between Cirrus and Github Actions.

Thank you for working on this!

> But finally, I managed to get a green build for all systems that we
> support on Cirrus CI!
>
> IMPORTANT CONTEXT: I only did a cursory review of the workflow file that
> Claude created. So there's probably still a bunch of cleanup to do, but
> I at least wanted to share this initial base. I don't know if I'll have
> any more time to work on this before June 1st. I have a newborn that's
> taking up a lot of my spare time. So, I'd be very happy if someone else
> wants to take this patch over and get it to a committable state.

I am actually working on the same thing and I was about to create a
new thread, I am glad that I checked my emails before doing that. I
can take this patch and move forward.

I am attaching my WIP version for visibility. My version doesn't have
BSD operating systems and uses Ubuntu instead of Debian. I didn't know
that containers can be used like you did. Another difference between
our versions is that mine has helper scripts to call things like
installing dependencies, configure, build and test. I did it that way
as we can use these helper scripts for other CI providers in the
future.

I think we can merge these two patches and move forward that way. I am
planning to review your patch and see what I can come up with to get
it to a committable state.

> A few things (apart from more extensive review) that I think should be
> improved soon (but maybe not before the first commit):
> 1. io_uring support is disabled. I couldn't get it to work on the GitHub
>    Actions runners, I think it's disabled in the host kernel. @Andres

I managed to get it to work. I think it was due to memlock and I
solved it by running this command: 'sudo prlimit --pid $$
--memlock=unlimited:unlimited' before running the tests.

> 2. It's not using pre-built images at the moment, except for the Linux
>    docker images I think. So it re-downloads a bunch of dependencies for
>    every build for most OSes. @Bilal or @Andres

Yes, that is a problem and needs to be handled separately.

One other problem is the logs. Github Actions' logs are not public,
you need to sign in to see them. I think we can solve them by
uploading logs to the public when the task fails but perhaps that can
be handled afterwards.

--
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-05-25T12:14:41Z

Hi,

We had an unconference session at the PGConf.dev 2026. Here are the
notes [1] (Thank you Lukas for taking and publishing the notes!). Some
important points are:

- We can use Github Actions for now, and revisit alternatives later.
- We won't have BSDs for the first version.
- Backpatch until PG15, where CI introduced.
- No need to test VS 2019, we can continue with VS 2022.
- Deal with making logs public later.

On Tue, 19 May 2026 at 01:27, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> I think we can merge these two patches and move forward that way. I am
> planning to review your patch and see what I can come up with to get
> it to a committable state.

Here is the v2, I took Jelte's patch and reviewed & merged it with my
patch. Updates and questions are:

1- I continued to use Jelte's container method (Linux tasks only for
now, BSD tasks will be included in the future) because I think that is
the future-proof way since we might want to generate our container
images in the future. Also, up-to-date Debian images can be tested
with this way; otherwise we would need to use Ubuntu 24.04.

2- io_uring tests work on the Linux Meson task.

3- I didn't put commands to helper scripts for now. I think it is a
good thing to have a helper script but it would be better to have this
helper script after the first version is committed since it can extend
the timeline. Also, I found that having all commands in one file makes
debugging easier.

4- FreeBSD task has these options:

      PG_TEST_INITDB_EXTRA_OPTS: >-
        -c debug_copy_parse_plan_trees=on
        -c debug_write_read_parse_plan_trees=on
        -c debug_raw_expression_coverage_test=on
        -c debug_parallel_query=regress

Since we won't have FreeBSD for the first version. I put these options
to the MacOS task but I couldn't decide where to put
'PG_TEST_PG_UPGRADE_MODE: --link'.

Also, I am planning to work on back patches when we agree on the
upstream one. Does that sound good?

CI run link of attached patch:
https://github.com/nbyavuz/postgres/actions/runs/26398508250

[1] https://wiki.postgresql.org/wiki/PGConf.dev_2026_Developer_Unconference

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Thomas Munro <thomas.munro@gmail.com> — 2026-05-27T03:54:19Z

On Tue, May 19, 2026 at 9:22 AM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
> 3. There's currently no integration with the CFBot or commitfest yet. @Thomas

Looking into this part urgently...

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-05-27T18:10:46Z

Hi,

> Here is the v2, I took Jelte's patch and reviewed & merged it with my
> patch. Updates and questions are:
> 
> 1- I continued to use Jelte's container method (Linux tasks only for
> now, BSD tasks will be included in the future) because I think that is
> the future-proof way since we might want to generate our container
> images in the future. Also, up-to-date Debian images can be tested
> with this way; otherwise we would need to use Ubuntu 24.04.

Good.


> 2- io_uring tests work on the Linux Meson task.

Is there a reason to not just do that for all the tasks?


> 3- I didn't put commands to helper scripts for now. I think it is a
> good thing to have a helper script but it would be better to have this
> helper script after the first version is committed since it can extend
> the timeline. Also, I found that having all commands in one file makes
> debugging easier.

Hm. I'm a bit worried about this getting pretty unmaintainable, due to the
repetition.  I think at least we need to use yaml anchors to deduplicate some
steps.


> 4- FreeBSD task has these options:
> 
>       PG_TEST_INITDB_EXTRA_OPTS: >-
>         -c debug_copy_parse_plan_trees=on
>         -c debug_write_read_parse_plan_trees=on
>         -c debug_raw_expression_coverage_test=on
>         -c debug_parallel_query=regress
> 
> Since we won't have FreeBSD for the first version. I put these options
> to the MacOS task but I couldn't decide where to put
> 'PG_TEST_PG_UPGRADE_MODE: --link'.

Makes sense.


> Also, I am planning to work on back patches when we agree on the
> upstream one. Does that sound good?

Yep.



> diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
> new file mode 100644
> index 00000000000..6d20068727c
> --- /dev/null
> +++ b/.github/workflows/ci.yml
> @@ -0,0 +1,1125 @@
> +# GitHub Actions CI configuration for PostgreSQL
> +
> +name: Github Actions CI
> +
> +on:
> +  push:
> +    branches: [ "*" ]
> +
> +# Default to the minimum privilege the jobs need (just reading the repo
> +# contents during checkout). Individual jobs override this when they need
> +# more, e.g. `cancel-previous` needs `actions: write` to cancel runs.
> +permissions:
> +  contents: read

I'm not sure I like that we ever need more than that. I'd expect that
postgresql-cfbot will explicitly disable write permissions for runs.


> +# NB: intentionally NO workflow-level `concurrency:` block. The native
> +# concurrency mechanism makes a new run wait for the previous one to fully
> +# cancel before it starts — which can take a while. Instead the
> +# `cancel-previous` job below fires a cancel API call asynchronously,
> +# so the new run gets going immediately. On master the cancel job is skipped,
> +# so every push runs to completion.

Is this really worth having our own code? Seems like it'd not be that frequent
to push if there are already running runs?  What kind of delays are we talking
about?




> +  # To avoid unnecessarily spinning up a lot of VMs / containers for entirely
> +  # broken commits, have a minimal task that all others depend on.
> +  #
> +  # SPECIAL:
> +  # - Builds with --auto-features=disabled and thus almost no enabled
> +  #   dependencies
> +  sanity-check:
> +    name: SanityCheck
> +    needs: setup
> +    if: needs.setup.outputs.sanitycheck == 'true'
> +    runs-on: ubuntu-latest
> +    timeout-minutes: 15
> +    container:
> +      image: ${{ needs.setup.outputs.linux_ci_image }}
> +    env:
> +      BUILD_JOBS: 8
> +      TEST_JOBS: 8
> +      CCACHE_DIR: ${{ github.workspace }}/ccache_dir
> +      # no options enabled, should be small
> +      CCACHE_MAXSIZE: "150M"
> +    steps:
> +      - uses: actions/checkout@v6
> +        with:
> +          fetch-depth: ${{ env.CLONE_DEPTH }}
> +
> +      - name: Restore ccache
> +        uses: actions/cache@v5

Seems like this is used by every task. Can we move this into a yaml anchor or
such, by using a variable representing the job name?


> +        with:
> +          path: ${{ env.CCACHE_DIR }}
> +          key: ccache-sanitycheck-${{ github.run_id }}
> +          restore-keys: ccache-sanitycheck-

Why is the key here the run id? Doesn't that mean that we will never have a
precise cache match and that we will keep multiple versions of the cache
around? That seems like a waste of cache space?

For efficiency, particularly on cfbot, it seems like it could be useful to
populate the cache of branches with the cache of the master branch. For that
we'd need the branch name in the key. Which I think would also good for
postgres/postgres, as we currently have a lot of interference between runs on
the main and the REL_XY_STABLE branches.


> +      - name: Prepare workspace
> +        run: |
> +          whoami
> +          useradd -m postgres
> +          chown -R postgres:postgres .
> +          mkdir -p "$CCACHE_DIR"
> +          chown -R postgres:postgres "$CCACHE_DIR"
> +          # Can't change the container's kernel.core_pattern; the postgres
> +          # user can't write to / normally. Make / writable.
> +          chown root:postgres /
> +          chmod g+rwx /

Why not just always use a privileged container?


> +      - name: Configure
> +        run: |
> +          su postgres <<-'EOF'
> +            set -e
> +            meson setup \
> +              --buildtype=debug \
> +              --auto-features=disabled \
> +              -Ddefault_library=shared \
> +              -Dtap_tests=enabled \
> +              build
> +          EOF
> +
> +      - name: Build
> +        run: |
> +          su postgres <<EOF
> +            set -e
> +            ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET}
> +          EOF

Should we have an explicit cache upload step here? Or are upload steps run
unconditionally?


> +      # Run a minimal set of tests. The main regression tests take too long
> +      # for this purpose. For now this is a random quick pg_regress style
> +      # test, and a tap test that exercises both a frontend binary and the
> +      # backend.
> +      - name: Test
> +        run: |
> +          su postgres <<EOF
> +            set -e
> +            ulimit -c unlimited
> +            meson test ${MTEST_ARGS} --suite setup
> +            meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \
> +              cube/regress pg_ctl/001_start_stop
> +          EOF
> +
> +      - name: Core backtraces
> +        if: failure()
> +        run: |
> +          mkdir -m 770 /tmp/cores
> +          find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \;
> +          src/tools/ci/cores_backtrace.sh linux /tmp/cores
> +
> +      - name: Upload logs
> +        if: failure()
> +        uses: actions/upload-artifact@v7
> +        with:
> +          name: sanitycheck-logs-${{ github.run_id }}
> +          path: |
> +            build*/testrun/**/*.log
> +            build*/testrun/**/*.diffs
> +            build*/testrun/**/regress_log_*
> +            build*/meson-logs/*.txt
> +          if-no-files-found: ignore

I think this really should be in a yaml anchor, we have a few somewhat
different versions of this now.

It's pretty annoying that the output of the failures isn't visible in the UI.
Maybe we ought to print a few of the failures out or something?


> +
> +  # SPECIAL:
> +  # - Uses address sanitizer (sanitizer failures are typically printed in
> +  #   the server log)
> +  # - Configures postgres with a small segment size
> +  #
> +  # Enable a reasonable set of sanitizers. Use the linux task for that, as
> +  # it's one of the fastest tasks (without sanitizers). Also several of the
> +  # sanitizers work best on linux.
> +  #
> +  # The overhead of alignment sanitizer is low, undefined behaviour has
> +  # moderate overhead. Test alignment sanitizer in the meson task, as it
> +  # does both 32 and 64 bit builds and is thus more likely to expose
> +  # alignment bugs.
> +  #
> +  # Address sanitizer in contrast is somewhat expensive. Enable it in the
> +  # autoconf task, as the meson task tests both 32 and 64bit.

I wonder if we should split the meson task into two, one for 32bit and one for
64bit. The concurrency limits for public repos are high enough for that to
seem like a reasonable tradeoff? There's no work, other than the repo
checkout, shared between them.


> +  # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes
> +  # print_stacktraces=1,verbosity=2, duh
> +  # detect_leaks=0: too many uninteresting leak errors in short-lived binaries
> +  linux-autoconf:
> +    name: Linux - Debian Trixie - Autoconf
> +    needs: [setup, sanity-check]
> +    if: |
> +      !cancelled() &&
> +      needs.setup.outputs.linux == 'true' &&
> +      needs.sanity-check.result != 'failure'
> +    runs-on: ubuntu-latest
> +    timeout-minutes: 60
> +    container:
> +      image: ${{ needs.setup.outputs.linux_ci_image }}
> +      # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates,
> +      # kill9's, and restarts postgres; with the container's small PID
> +      # space a new postgres can recycle the dead postmaster's PID before
> +      # pg_ctl's postmaster.pid check notices, producing spurious "node X
> +      # is already running" failures. SysV shm in the test also relies on
> +      # host-like IPC behavior.
> +      #
> +      # --ulimit raises memlock and core dump size. Memlock is needed for
> +      # running the AIO tests.
> +      #
> +      # --privileged is needed so the prepare step can write to sysctls
> +      # under /proc/sys (it's mounted read-only without it). We use it to
> +      # set kernel.core_pattern.
> +      options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged
> +    env:
> +      BUILD_JOBS: 4
> +      TEST_JOBS: 8
> +      CCACHE_DIR: /tmp/ccache_dir
> +      DEBUGINFOD_URLS: "https://debuginfod.debian.net"
> +
> +      SANITIZER_FLAGS: -fsanitize=address
> +      UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2
> +      ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0
> +      CFLAGS: -Og -ggdb -fno-sanitize-recover=all -fsanitize=address
> +      CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all -fsanitize=address
> +      LDFLAGS: -fsanitize=address
> +      CC: ccache gcc
> +      CXX: ccache g++

There's a fair bit of stuff shared between the meson/autoconf linux
tasks. Previously they used a matrix to reduce that a *bit*. But now it's
entirely duplicated, including stuff that doesn't apply to the current job
(e.g. UBSAN_OPTIONS/ASAN_OPTIONS).  And blocks like the following:


> +      - name: Prepare workspace
> +        run: |
> +          useradd -m postgres
> +          chown -R postgres:postgres .
> +          mkdir -p "$CCACHE_DIR"
> +          chown -R postgres:postgres "$CCACHE_DIR"
> +          mkdir -m 770 /tmp/cores
> +          chown root:postgres /tmp/cores
> +          sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core'
> +
> +          # Hosts for the load balance test
> +          cat >> /etc/hosts <<-EOF
> +            127.0.0.1 pg-loadbalancetest
> +            127.0.0.2 pg-loadbalancetest
> +            127.0.0.3 pg-loadbalancetest
> +          EOF


> +      # Install dependencies via Homebrew rather than Macports. On stock
> +      # GH runners macports requires a heavy bootstrap, and the relevant
> +      # Postgres deps are all available in brew.

What does "heavy bootstrap" mean?

> +      - name: Install dependencies
> +        run: |
> +          brew update
> +          brew install \
> +            ccache meson openldap python@3.12 tcl-tk
> +          # IPC::Run via cpanm (system perl)
> +          sudo cpan -T -i IPC::Run IO::Tty

We do spend ~95s on this every run, that's not nothing. And it puts a bunch of
load onto the brew's mirrors to do that every run.


> +      - name: Test world
> +        run: |
> +          ulimit -c unlimited
> +          ulimit -n 1024
> +          meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS}

I'd re-add the comments that were in .cirrus.yml about this.


> +  windows-vs:
> +    name: Windows - Server 2022, VS 2022 - Meson & ninja
> +    needs: [setup, sanity-check]
> +    if: |
> +      !cancelled() &&
> +      needs.setup.outputs.windows == 'true' &&
> +      needs.sanity-check.result != 'failure'
> +    runs-on: windows-2022
> +    timeout-minutes: 60
> +    env:
> +      TEST_JOBS: 8
> +      # Avoid port conflicts between concurrent tap tests
> +      PG_TEST_USE_UNIX_SOCKETS: 1
> +      PG_REGRESS_SOCK_DIR: 'c:\pgsock\'

At least my editor gets confused by the \', thinking it's escaping the '. As
everything just works without the trailing \, I'd go that way.



> +      # The TAP tests build an initdb template under build/tmp_install and
> +      # then `robocopy` it into per-test data directories. Robocopy with the
> +      # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from
> +      # their parent dir. On GitHub-hosted Windows runners the workspace's
> +      # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT
> +      # grant the runner user (runneradmin) directly. That matters because
> +      # pg_ctl on Windows uses CreateRestrictedProcess to drop admin
> +      # privileges from postmaster, so the postmaster process has the user
> +      # SID in its token but no longer the Administrators group — leaving it
> +      # with only "Users:(RX)" on pg_control and friends, which causes
> +      # "PANIC: could not open file global/pg_control: Permission denied".
> +      #
> +      # Fix it once on the workspace dir with (OI)(CI) inheritance flags so
> +      # every file/dir created underneath gets an explicit grant for the
> +      # current user.
> +      - name: Grant workspace ACL to runner user
> +        shell: pwsh
> +        run: |
> +          icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null
> +          Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}"

Perhaps this would be better to fix by changing the robocopy flags?


> +      # postgres' plpython3u loads python3.dll (the stable-ABI forwarder)
> +      # which in turn loads whichever python3NN.dll the Windows loader finds
> +      # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships
> +      # its own python3.dll + python39.dll and appears on PATH *before* the
> +      # hostedtoolcache Python 3.12 — so without intervention the backend
> +      # ends up running Python 3.9 while postgres' stdlib search uses 3.12,
> +      # producing `ImportError: cannot import name 'text_encoding' from
> +      # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`).
> +      #
> +      # Pin PYTHONHOME to the Python 3.12 prefix, and prepend that prefix
> +      # to PATH so its python3.dll wins the DLL search.
> +      - name: Pin Python prefix on PATH and PYTHONHOME
> +        shell: pwsh
> +        run: |
> +          $prefix = (python -c "import sys; print(sys.prefix)").Trim()
> +          Add-Content $env:GITHUB_ENV "PYTHONHOME=$prefix"
> +          Add-Content $env:GITHUB_PATH $prefix
> +          Write-Host "PYTHONHOME=$prefix"
> +          Write-Host "Prepended $prefix to PATH"

GRJGJKLJKJDFJKDF.


> +      - name: Install dependencies
> +        shell: pwsh
> +        run: |
> +          choco install -y --no-progress --limitoutput diffutils winflexbison
> +          # meson + ninja aren't preinstalled on windows-2022. Install via pip
> +          python -m pip install --upgrade meson ninja
> +
> +          # OpenSSL 1.1 via the slproweb installer (pinned to match the
> +          # version used elsewhere in postgres CI).
> +          curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe
> +          Start-Process -Wait -FilePath ./openssl-setup.exe `
> +            -ArgumentList '/DIR=c:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES'
> +          # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll
> +          # in c:\openssl\1.1\bin\ and updates the system PATH. GH Actions
> +          # snapshots PATH at job start though, so the running job won't
> +          # see those DLLs and initdb.exe would crash silently at runtime.
> +          # Push the bin dir onto GITHUB_PATH so it persists for later steps.
> +          Add-Content $env:GITHUB_PATH "c:\openssl\1.1\bin"

I don't like that much, but I'm not sure we have a better alternative
short-term.
> +  windows-mingw:
> +    name: Windows - Server 2022, MinGW64 - Meson
> +    needs: [setup, sanity-check]
> +    if: |
> +      !cancelled() &&
> +      needs.setup.outputs.mingw == 'true' &&
> +      needs.sanity-check.result != 'failure'
> +    runs-on: windows-2022
> +    timeout-minutes: 60
> +    env:
> +      TEST_JOBS: 4  # higher concurrency causes occasional failures
> +      PG_TEST_USE_UNIX_SOCKETS: 1
> +      PG_REGRESS_SOCK_DIR: 'c:\pgsock\'
> +      TAR: "c:/windows/system32/tar.exe"
> +      # for mingw plpython to find its installation
> +      PYTHONHOME: D:/a/_temp/msys64/ucrt64
> +
> +      MSYS: winjitdebug
> +      CHERE_INVOKING: 1
> +      MESON_FEATURES: >-
> +        -Dnls=disabled

Missing comments from .cirrus.tasks.yml


Thanks for working on this!

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Jelte Fennema <postgres@jeltef.nl> — 2026-05-27T20:33:30Z

I didn't look at the patch, and I won't be able too before June 1st.
But I wanted to give some quick context on the things andres called
out, that I'm pretty sure originate from the patch I submitted.

On Wed, 27 May 2026 at 20:10, Andres Freund <andres@anarazel.de> wrote:
> > +# NB: intentionally NO workflow-level `concurrency:` block. The native
> > +# concurrency mechanism makes a new run wait for the previous one to fully
> > +# cancel before it starts — which can take a while. Instead the
> > +# `cancel-previous` job below fires a cancel API call asynchronously,
> > +# so the new run gets going immediately. On master the cancel job is skipped,
> > +# so every push runs to completion.
>
> Is this really worth having our own code? Seems like it'd not be that frequent
> to push if there are already running runs?  What kind of delays are we talking
> about?

I agree this doesn't pull its weight and can be removed. It was part
of me trying to iterate quickly. I think it could take a few minutes
to cancel some of the jobs BSD nested virtualized jobs (might have
been other jobs though).

> I wonder if we should split the meson task into two, one for 32bit and one for
> 64bit.

+1

> > +      # Install dependencies via Homebrew rather than Macports. On stock
> > +      # GH runners macports requires a heavy bootstrap, and the relevant
> > +      # Postgres deps are all available in brew.
>
> What does "heavy bootstrap" mean?

Not sure. This was Claude's doing. MacOS was green pretty quickly, so
I didn't bother questioning details while all the other builds were
still red.

> We do spend ~95s on this every run, that's not nothing. And it puts a bunch of
> load onto the brew's mirrors to do that every run.

I think we can only avoid that if we have our own runners.

> > +      # The TAP tests build an initdb template under build/tmp_install and
> > +      # then `robocopy` it into per-test data directories. Robocopy with the
> > +      # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from
> > +      # their parent dir. On GitHub-hosted Windows runners the workspace's
> > +      # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT
> > +      # grant the runner user (runneradmin) directly. That matters because
> > +      # pg_ctl on Windows uses CreateRestrictedProcess to drop admin
> > +      # privileges from postmaster, so the postmaster process has the user
> > +      # SID in its token but no longer the Administrators group — leaving it
> > +      # with only "Users:(RX)" on pg_control and friends, which causes
> > +      # "PANIC: could not open file global/pg_control: Permission denied".
> > +      #
> > +      # Fix it once on the workspace dir with (OI)(CI) inheritance flags so
> > +      # every file/dir created underneath gets an explicit grant for the
> > +      # current user.
> > +      - name: Grant workspace ACL to runner user
> > +        shell: pwsh
> > +        run: |
> > +          icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null
> > +          Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}"
>
> Perhaps this would be better to fix by changing the robocopy flags?

Getting the Windows build working took a lot of work. To be clear that
involved me copy-pasting log output into Claude or pointing it to
download log tarballs. All of these huge comments I *did not* write.
While iterating the comments seemed believable (but LLMs are good at
that). My intent was to review them for correctness and for cleaner
solutions. But I did not have time nor energy for that anymore. So
yeah other fixes might very well be better (similarly for the
python3 or openssl stuff).

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-05-27T22:15:46Z

On Wed, May 27, 2026 at 11:10 AM Andres Freund <andres@anarazel.de> wrote:
> > +# Default to the minimum privilege the jobs need (just reading the repo
> > +# contents during checkout). Individual jobs override this when they need
> > +# more, e.g. `cancel-previous` needs `actions: write` to cancel runs.
> > +permissions:
> > +  contents: read
>
> I'm not sure I like that we ever need more than that. I'd expect that
> postgresql-cfbot will explicitly disable write permissions for runs.

+1, and +1 for getting rid of the custom cancel, for that reason.

- Do we need to defend our downstream forks from this workflow? (We
have 5,700 of them, apparently.)
- Do the pginfra folks who own the repo need to lock down all the
Actions settings before we ship this? (On my fork, at least, the
default settings were horrifically permissive.)

--Jacob

Re: Heads Up: cirrus-ci is shutting down June 1st

Zsolt Parragi <zsolt.parragi@percona.com> — 2026-05-27T23:10:44Z

Hello!

I didn't try the workflow on github yet, I only have a few comments
based on the yaml file:

+on:
+  push:
+    branches: [ "*" ]
+

Should this be "**", otherwise we ignore branches with a "/" in it?

+      SANITIZER_FLAGS: -fsanitize=address

is this used anywhere? Seems like it's directly included into CFLAGS/LDFLAGS.

Re: Heads Up: cirrus-ci is shutting down June 1st

Peter Eisentraut <peter@eisentraut.org> — 2026-05-28T11:49:10Z

On 25.05.26 14:14, Nazir Bilal Yavuz wrote:
> On Tue, 19 May 2026 at 01:27, Nazir Bilal Yavuz<byavuz81@gmail.com> wrote:
>> I think we can merge these two patches and move forward that way. I am
>> planning to review your patch and see what I can come up with to get
>> it to a committable state.
> Here is the v2, I took Jelte's patch and reviewed & merged it with my
> patch.

I have tested this patch and inspected the output mostly to make sure 
that there are no regressions about what features and dependency 
versions are being tested (diffed the various logs).  I'm proposing a 
few minor fixups in the attached patch, but other than that (and what 
others have mentioned), this pretty much works well, and I would be 
content to proceed with this or whatever state it's on in a few days.

Some comments in detail:

- Others have already mentioned about the potential for this to conflict 
with downstream uses of GH Actions.  I suggest renaming the file from 
ci.yml to something like postgresql-ci.yml, so that there is no file 
naming conflict or confusion.

- As was already mentioned, the Linux/Meson job is very long (slow) and 
should be split into separate 32/64-bit jobs.

- The job names are too long and get truncated in the UI.  This is 
especially annoying when the important differentiator like "Autoconf" or 
"Meson" gets cut off.  I'm proposing some changes to the job names that 
make them display better.  (Also consider this if you make separate jobs 
for 32/64-bit.  The usable space is about 20 characters.)

- On macOS, there were some dependency differences:

   - readline was not used.
   - tcl-tk (version 9) was used instead of tcl-tk@8.
   - python@3.12 was installed but not actually used in the build.
   - zlib version differed.

   Maybe the zlib difference is not important and could be ignored.
   Also, maybe we don't need to use a versioned python dependency.  (We
   didn't have one before we switched Cirrus from Homebrew to MacPorts.)

- On macOS, the meson setup output reported a significantly different 
sysroot, which was confusing.  I think the sysroot is only used if you 
build against a system perl/python/tcl, which we don't, so I added an 
option to disable the sysroot use.  That way, if we do end up making use 
of the sysroot, someone is forced to investigate this issue.  I don't 
know if this makes sense.

- For macOS, I threw in some HOMEBREW_* environment variables to disable 
some unnecessary additional output or cleanup steps.

- On Windows/VS, we should install winflexbison3 not winflexbison, to 
get an up-to-date version.

- FUTURE: On Windows/VS, we use openssl 1.1, which matches the Cirrus 
setup, so it's ok, but the equivalent buildfarm members all use openssl 
3.*, so we should consider upgrading that sometime to make that more 
consistent.

- On Windows/minGW, I dropped a few packages from the set to be 
installed, which didn't seem necessary.

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-05-28T15:07:22Z

Hi,

On 2026-05-27 15:15:46 -0700, Jacob Champion wrote:
> On Wed, May 27, 2026 at 11:10 AM Andres Freund <andres@anarazel.de> wrote:
> > > +# Default to the minimum privilege the jobs need (just reading the repo
> > > +# contents during checkout). Individual jobs override this when they need
> > > +# more, e.g. `cancel-previous` needs `actions: write` to cancel runs.
> > > +permissions:
> > > +  contents: read
> >
> > I'm not sure I like that we ever need more than that. I'd expect that
> > postgresql-cfbot will explicitly disable write permissions for runs.
> 
> +1, and +1 for getting rid of the custom cancel, for that reason.
> 
> - Do we need to defend our downstream forks from this workflow? (We
> have 5,700 of them, apparently.)

I don't see why. I think it's good if they run CI. Having forks not run CI by
default would imo take one of the main advantages of using github actions
away.


> - Do the pginfra folks who own the repo need to lock down all the
> Actions settings before we ship this? (On my fork, at least, the
> default settings were horrifically permissive.)

Yes, they are too permissive by default, including on postgres/postgres.  I
think postgres/postgres isn't *that* threatened, but we should make things are
shored up anyway. Where it's really crucial is the postgresql-cfbot repo.

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-05-28T15:51:09Z

On Thu, May 28, 2026 at 8:07 AM Andres Freund <andres@anarazel.de> wrote:
> On 2026-05-27 15:15:46 -0700, Jacob Champion wrote:
> > - Do we need to defend our downstream forks from this workflow? (We
> > have 5,700 of them, apparently.)
>
> I don't see why. I think it's good if they run CI. Having forks not run CI by
> default would imo take one of the main advantages of using github actions
> away.

I was imagining a quick opt-in, like the Cirrus flow did, that fork
owners can do once they have checked their settings.

(I thought we planned to research medium-term alternatives to Actions
anyway; is it important that the entire graph starts running hundreds
or thousands of CI copies right away?)

> Yes, they are too permissive by default, including on postgres/postgres.  I
> think postgres/postgres isn't *that* threatened, but we should make things are
> shored up anyway. Where it's really crucial is the postgresql-cfbot repo.

Combining with the above: I'm worried that if all of our 5.7k forks
have permissive settings, and we accidentally ship a workflow
vulnerability that doesn't affect us but does affect them, that would
not be a fun cleanup.

--Jacob

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-05-28T16:13:07Z

Hi,

On 2026-05-28 08:51:09 -0700, Jacob Champion wrote:
> On Thu, May 28, 2026 at 8:07 AM Andres Freund <andres@anarazel.de> wrote:
> > On 2026-05-27 15:15:46 -0700, Jacob Champion wrote:
> > > - Do we need to defend our downstream forks from this workflow? (We
> > > have 5,700 of them, apparently.)
> >
> > I don't see why. I think it's good if they run CI. Having forks not run CI by
> > default would imo take one of the main advantages of using github actions
> > away.
> 
> I was imagining a quick opt-in, like the Cirrus flow did, that fork
> owners can do once they have checked their settings.

I'm not aware of a good way to do that.  I'm sure we could hack up a way,
e.g. by requiring an environment variable to be configured on the repo level
to opt-in, but it seems pretty crufty.

I think making it easier for forks to run CI is a far bigger gain than the
risk of GHA doing something stupid in a fork.  There were a lot of folks that
didn't realize that they could run CI individually or had a hard time enabling
it.

> (I thought we planned to research medium-term alternatives to Actions
> anyway; is it important that the entire graph starts running hundreds
> or thousands of CI copies right away?)

I suspect where we will end up coming out is that we use an alternative for
actions for cfbot and regular contributors, but that everyone else will use
GHA.

> > Yes, they are too permissive by default, including on postgres/postgres.
> > I think postgres/postgres isn't *that* threatened, but we should make
> > things are shored up anyway. Where it's really crucial is the
> > postgresql-cfbot repo.
> 
> Combining with the above: I'm worried that if all of our 5.7k forks have
> permissive settings, and we accidentally ship a workflow vulnerability that
> doesn't affect us but does affect them, that would not be a fun cleanup.

I'm not sure what path for that would exist that don't already? ISTM that'd
require downstream repos to have added their own actions workflows that
somehow interact with ours, or that they blindly run PRs from unknown folks
that add new workflows - in either case it seems they have a problem
independent of us shipping a runs-by-default actions workflow?

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-05-28T16:19:15Z

Hi,

On 2026-05-25 15:14:41 +0300, Nazir Bilal Yavuz wrote:
> Also, I am planning to work on back patches when we agree on the
> upstream one. Does that sound good?

On this aspect: I suspect that a sane process here might be to merge GHA CI
into master only, run cfbot for a few days, and only then backpatch the
support to the older branches.  It seems rather likely that there will be some
stability improvements we'll have to do initially, and that we only will find
those issues with sufficient runs.  If we have to backpatch all of those it'll
be a lot more work (and noise).

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-05-28T17:04:51Z

On Thu, May 28, 2026 at 9:13 AM Andres Freund <andres@anarazel.de> wrote:
> On 2026-05-28 08:51:09 -0700, Jacob Champion wrote:
> > I was imagining a quick opt-in, like the Cirrus flow did, that fork
> > owners can do once they have checked their settings.
>
> I'm not aware of a good way to do that.  I'm sure we could hack up a way,
> e.g. by requiring an environment variable to be configured on the repo level
> to opt-in,

That was more or less my thought.

> but it seems pretty crufty.
>
> I think making it easier for forks to run CI is a far bigger gain than the
> risk of GHA doing something stupid in a fork.  There were a lot of folks that
> didn't realize that they could run CI individually or had a hard time enabling
> it.

Right, but Cirrus only ever had the ability to run a CI, not write to
the code base it was running. If we unleash a bunch of newcomer GitHub
CIs without first explaining "hey, you really need to lock some stuff
down first", I think we may be doing them all a disservice.

Especially since GitHub claims to protect downstream forks from this
[1] -- which is undocumented? -- but that protection appears to not
actually work [2] if we push a workflow at the root of the graph. (I
haven't verified any of that myself yet, but in the absence of
documentation, I'm not really optimistic.)

> > Combining with the above: I'm worried that if all of our 5.7k forks have
> > permissive settings, and we accidentally ship a workflow vulnerability that
> > doesn't affect us but does affect them, that would not be a fun cleanup.
>
> I'm not sure what path for that would exist that don't already?

Using the current v2 patch, for instance, a `actions: write` token
that gets leaked by accident can then be used to approve pending
workflow runs. (Consensus seems to be forming that we shouldn't have
those privileges in the workflow spec, but we have to all remember why
that rule exists when we're reviewing workflow patches.)

--Jacob

[1] https://github.com/github/docs/issues/15761
[2] https://github.com/orgs/community/discussions/53510

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-05-28T17:06:22Z

Hi,

Thank you for looking into this!

On Wed, 27 May 2026 at 21:10, Andres Freund <andres@anarazel.de> wrote:
>
> > Here is the v2, I took Jelte's patch and reviewed & merged it with my
> > patch. Updates and questions are:
> >
> > 1- I continued to use Jelte's container method (Linux tasks only for
> > now, BSD tasks will be included in the future) because I think that is
> > the future-proof way since we might want to generate our container
> > images in the future. Also, up-to-date Debian images can be tested
> > with this way; otherwise we would need to use Ubuntu 24.04.
>
> Good.
>
>
> > 2- io_uring tests work on the Linux Meson task.
>
> Is there a reason to not just do that for all the tasks?

I might word it incorrectly. I meant that Linux meson tests use:

PG_TEST_INITDB_EXTRA_OPTS: >-
  -c io_method=io_uring

and that wasn't working before, now it works. I guess we have this
only on Linux because we wanted to test io_method=worker in the other
tasks.


> > 3- I didn't put commands to helper scripts for now. I think it is a
> > good thing to have a helper script but it would be better to have this
> > helper script after the first version is committed since it can extend
> > the timeline. Also, I found that having all commands in one file makes
> > debugging easier.
>
> Hm. I'm a bit worried about this getting pretty unmaintainable, due to the
> repetition.  I think at least we need to use yaml anchors to deduplicate some
> steps.

Github Actions added support of yaml anchors last year but
unfortunately they don't support merge keys. Related information: [1].


> > 4- FreeBSD task has these options:
> >
> >       PG_TEST_INITDB_EXTRA_OPTS: >-
> >         -c debug_copy_parse_plan_trees=on
> >         -c debug_write_read_parse_plan_trees=on
> >         -c debug_raw_expression_coverage_test=on
> >         -c debug_parallel_query=regress
> >
> > Since we won't have FreeBSD for the first version. I put these options
> > to the MacOS task but I couldn't decide where to put
> > 'PG_TEST_PG_UPGRADE_MODE: --link'.
>
> Makes sense.
>
>
> > Also, I am planning to work on back patches when we agree on the
> > upstream one. Does that sound good?
>
> Yep.
>
>
>
> > diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
> > new file mode 100644
> > index 00000000000..6d20068727c
> > --- /dev/null
> > +++ b/.github/workflows/ci.yml
> > @@ -0,0 +1,1125 @@
> > +# GitHub Actions CI configuration for PostgreSQL
> > +
> > +name: Github Actions CI
> > +
> > +on:
> > +  push:
> > +    branches: [ "*" ]
> > +
> > +# Default to the minimum privilege the jobs need (just reading the repo
> > +# contents during checkout). Individual jobs override this when they need
> > +# more, e.g. `cancel-previous` needs `actions: write` to cancel runs.
> > +permissions:
> > +  contents: read
>
> I'm not sure I like that we ever need more than that. I'd expect that
> postgresql-cfbot will explicitly disable write permissions for runs.

Done. Updated the comment and removed the 'Cancel previous runs' step.


> > +# NB: intentionally NO workflow-level `concurrency:` block. The native
> > +# concurrency mechanism makes a new run wait for the previous one to fully
> > +# cancel before it starts — which can take a while. Instead the
> > +# `cancel-previous` job below fires a cancel API call asynchronously,
> > +# so the new run gets going immediately. On master the cancel job is skipped,
> > +# so every push runs to completion.
>
> Is this really worth having our own code? Seems like it'd not be that frequent
> to push if there are already running runs?  What kind of delays are we talking
> about?

Jelte already answered this in [2]. 'Cancel previous runs' step is
removed and concurrency is used instead.


> > +  # To avoid unnecessarily spinning up a lot of VMs / containers for entirely
> > +  # broken commits, have a minimal task that all others depend on.
> > +  #
> > +  # SPECIAL:
> > +  # - Builds with --auto-features=disabled and thus almost no enabled
> > +  #   dependencies
> > +  sanity-check:
> > +    name: SanityCheck
> > +    needs: setup
> > +    if: needs.setup.outputs.sanitycheck == 'true'
> > +    runs-on: ubuntu-latest
> > +    timeout-minutes: 15
> > +    container:
> > +      image: ${{ needs.setup.outputs.linux_ci_image }}
> > +    env:
> > +      BUILD_JOBS: 8
> > +      TEST_JOBS: 8
> > +      CCACHE_DIR: ${{ github.workspace }}/ccache_dir
> > +      # no options enabled, should be small
> > +      CCACHE_MAXSIZE: "150M"
> > +    steps:
> > +      - uses: actions/checkout@v6
> > +        with:
> > +          fetch-depth: ${{ env.CLONE_DEPTH }}
> > +
> > +      - name: Restore ccache
> > +        uses: actions/cache@v5
>
> Seems like this is used by every task. Can we move this into a yaml anchor or
> such, by using a variable representing the job name?

Github Actions doesn't support merge keys. So we can't really
duplicate them. I used yaml anchors for the checkout step since it is
exactly for all jobs.


> > +        with:
> > +          path: ${{ env.CCACHE_DIR }}
> > +          key: ccache-sanitycheck-${{ github.run_id }}
> > +          restore-keys: ccache-sanitycheck-
>
> Why is the key here the run id? Doesn't that mean that we will never have a
> precise cache match and that we will keep multiple versions of the cache
> around? That seems like a waste of cache space?
>
> For efficiency, particularly on cfbot, it seems like it could be useful to
> populate the cache of branches with the cache of the master branch. For that
> we'd need the branch name in the key. Which I think would also good for
> postgres/postgres, as we currently have a lot of interference between runs on
> the main and the REL_XY_STABLE branches.

I think that is the default way. If the cache has the exact hit, it
doesn't refresh the cache. So, having ${{ github.run_id }} makes sure
we won't have exact hits and the cache will always be refreshed. This
sounds bad but that is what I understood :(

I can implement something like this:

      - name: Restore ccache
        uses: actions/cache/restore@v5
        with:
          path: ${{ env.CCACHE_DIR }}
          key: ccache-sanitycheck-master
          restore-keys: |
            ccache-sanitycheck-${{ github.ref_name }}
            ccache-sanitycheck-

      - name: Save ccache
        if: always()
        uses: actions/cache/save@v5
        with:
          path: ${{ env.CCACHE_DIR }}
          key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }}

So, it will first look for master's cache, then current branch's cache
and lastly whatever cache is available. Do you prefer that?


> > +      - name: Prepare workspace
> > +        run: |
> > +          whoami
> > +          useradd -m postgres
> > +          chown -R postgres:postgres .
> > +          mkdir -p "$CCACHE_DIR"
> > +          chown -R postgres:postgres "$CCACHE_DIR"
> > +          # Can't change the container's kernel.core_pattern; the postgres
> > +          # user can't write to / normally. Make / writable.
> > +          chown root:postgres /
> > +          chmod g+rwx /
>
> Why not just always use a privileged container?

Done.


> > +      - name: Configure
> > +        run: |
> > +          su postgres <<-'EOF'
> > +            set -e
> > +            meson setup \
> > +              --buildtype=debug \
> > +              --auto-features=disabled \
> > +              -Ddefault_library=shared \
> > +              -Dtap_tests=enabled \
> > +              build
> > +          EOF
> > +
> > +      - name: Build
> > +        run: |
> > +          su postgres <<EOF
> > +            set -e
> > +            ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET}
> > +          EOF
>
> Should we have an explicit cache upload step here? Or are upload steps run
> unconditionally?

Like I explained above, that is done by having ${{ github.run_id }} in
the cache key.


> > +      # Run a minimal set of tests. The main regression tests take too long
> > +      # for this purpose. For now this is a random quick pg_regress style
> > +      # test, and a tap test that exercises both a frontend binary and the
> > +      # backend.
> > +      - name: Test
> > +        run: |
> > +          su postgres <<EOF
> > +            set -e
> > +            ulimit -c unlimited
> > +            meson test ${MTEST_ARGS} --suite setup
> > +            meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \
> > +              cube/regress pg_ctl/001_start_stop
> > +          EOF
> > +
> > +      - name: Core backtraces
> > +        if: failure()
> > +        run: |
> > +          mkdir -m 770 /tmp/cores
> > +          find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \;
> > +          src/tools/ci/cores_backtrace.sh linux /tmp/cores
> > +
> > +      - name: Upload logs
> > +        if: failure()
> > +        uses: actions/upload-artifact@v7
> > +        with:
> > +          name: sanitycheck-logs-${{ github.run_id }}
> > +          path: |
> > +            build*/testrun/**/*.log
> > +            build*/testrun/**/*.diffs
> > +            build*/testrun/**/regress_log_*
> > +            build*/meson-logs/*.txt
> > +          if-no-files-found: ignore
>
> I think this really should be in a yaml anchor, we have a few somewhat
> different versions of this now.

Same thing, we can't have yaml anchors because merge keys are not
supported.  I created this variable:

_LOG_PATHS: &log_paths |
build*/testrun/**/*.log
build*/testrun/**/*.diffs
build*/testrun/**/regress_log_*
build*/meson-logs/*.txt

and used it in the Upload logs' path.


> It's pretty annoying that the output of the failures isn't visible in the UI.
> Maybe we ought to print a few of the failures out or something?

We already have '--print-errorlogs', do you mean something different?


> > +
> > +  # SPECIAL:
> > +  # - Uses address sanitizer (sanitizer failures are typically printed in
> > +  #   the server log)
> > +  # - Configures postgres with a small segment size
> > +  #
> > +  # Enable a reasonable set of sanitizers. Use the linux task for that, as
> > +  # it's one of the fastest tasks (without sanitizers). Also several of the
> > +  # sanitizers work best on linux.
> > +  #
> > +  # The overhead of alignment sanitizer is low, undefined behaviour has
> > +  # moderate overhead. Test alignment sanitizer in the meson task, as it
> > +  # does both 32 and 64 bit builds and is thus more likely to expose
> > +  # alignment bugs.
> > +  #
> > +  # Address sanitizer in contrast is somewhat expensive. Enable it in the
> > +  # autoconf task, as the meson task tests both 32 and 64bit.
>
> I wonder if we should split the meson task into two, one for 32bit and one for
> 64bit. The concurrency limits for public repos are high enough for that to
> seem like a reasonable tradeoff? There's no work, other than the repo
> checkout, shared between them.

Done.


> > +  # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes
> > +  # print_stacktraces=1,verbosity=2, duh
> > +  # detect_leaks=0: too many uninteresting leak errors in short-lived binaries
> > +  linux-autoconf:
> > +    name: Linux - Debian Trixie - Autoconf
> > +    needs: [setup, sanity-check]
> > +    if: |
> > +      !cancelled() &&
> > +      needs.setup.outputs.linux == 'true' &&
> > +      needs.sanity-check.result != 'failure'
> > +    runs-on: ubuntu-latest
> > +    timeout-minutes: 60
> > +    container:
> > +      image: ${{ needs.setup.outputs.linux_ci_image }}
> > +      # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates,
> > +      # kill9's, and restarts postgres; with the container's small PID
> > +      # space a new postgres can recycle the dead postmaster's PID before
> > +      # pg_ctl's postmaster.pid check notices, producing spurious "node X
> > +      # is already running" failures. SysV shm in the test also relies on
> > +      # host-like IPC behavior.
> > +      #
> > +      # --ulimit raises memlock and core dump size. Memlock is needed for
> > +      # running the AIO tests.
> > +      #
> > +      # --privileged is needed so the prepare step can write to sysctls
> > +      # under /proc/sys (it's mounted read-only without it). We use it to
> > +      # set kernel.core_pattern.
> > +      options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged
> > +    env:
> > +      BUILD_JOBS: 4
> > +      TEST_JOBS: 8
> > +      CCACHE_DIR: /tmp/ccache_dir
> > +      DEBUGINFOD_URLS: "https://debuginfod.debian.net"
> > +
> > +      SANITIZER_FLAGS: -fsanitize=address
> > +      UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2
> > +      ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0
> > +      CFLAGS: -Og -ggdb -fno-sanitize-recover=all -fsanitize=address
> > +      CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all -fsanitize=address
> > +      LDFLAGS: -fsanitize=address
> > +      CC: ccache gcc
> > +      CXX: ccache g++
>
> There's a fair bit of stuff shared between the meson/autoconf linux
> tasks. Previously they used a matrix to reduce that a *bit*. But now it's
> entirely duplicated, including stuff that doesn't apply to the current job
> (e.g. UBSAN_OPTIONS/ASAN_OPTIONS).  And blocks like the following:
>
>
> > +      - name: Prepare workspace
> > +        run: |
> > +          useradd -m postgres
> > +          chown -R postgres:postgres .
> > +          mkdir -p "$CCACHE_DIR"
> > +          chown -R postgres:postgres "$CCACHE_DIR"
> > +          mkdir -m 770 /tmp/cores
> > +          chown root:postgres /tmp/cores
> > +          sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core'
> > +
> > +          # Hosts for the load balance test
> > +          cat >> /etc/hosts <<-EOF
> > +            127.0.0.1 pg-loadbalancetest
> > +            127.0.0.2 pg-loadbalancetest
> > +            127.0.0.3 pg-loadbalancetest
> > +          EOF


I found we can use matrices and merged all linux tasks. I am not sure
that is better since it is a bit harder to read now.


> > +      # Install dependencies via Homebrew rather than Macports. On stock
> > +      # GH runners macports requires a heavy bootstrap, and the relevant
> > +      # Postgres deps are all available in brew.
>
> What does "heavy bootstrap" mean?

I used MacPorts on my first version. It took ~10 minutes to download
MacPorts. I think that if we could use caching like we did in the
Cirrus, it makes sense to use MacPorts. I will spend some time on
that.

And after spending some time, I am able to make it work. Now the first
run's dependencies install takes ~10 minutes since there is no
MacPorts cache but subsequent runs' install only take ~5 seconds.


> > +      - name: Install dependencies
> > +        run: |
> > +          brew update
> > +          brew install \
> > +            ccache meson openldap python@3.12 tcl-tk
> > +          # IPC::Run via cpanm (system perl)
> > +          sudo cpan -T -i IPC::Run IO::Tty
>
> We do spend ~95s on this every run, that's not nothing. And it puts a bunch of
> load onto the brew's mirrors to do that every run.

You are right. MacPorts is used now.


> > +      - name: Test world
> > +        run: |
> > +          ulimit -c unlimited
> > +          ulimit -n 1024
> > +          meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS}
>
> I'd re-add the comments that were in .cirrus.yml about this.

Done.


> > +  windows-vs:
> > +    name: Windows - Server 2022, VS 2022 - Meson & ninja
> > +    needs: [setup, sanity-check]
> > +    if: |
> > +      !cancelled() &&
> > +      needs.setup.outputs.windows == 'true' &&
> > +      needs.sanity-check.result != 'failure'
> > +    runs-on: windows-2022
> > +    timeout-minutes: 60
> > +    env:
> > +      TEST_JOBS: 8
> > +      # Avoid port conflicts between concurrent tap tests
> > +      PG_TEST_USE_UNIX_SOCKETS: 1
> > +      PG_REGRESS_SOCK_DIR: 'c:\pgsock\'
>
> At least my editor gets confused by the \', thinking it's escaping the '. As
> everything just works without the trailing \, I'd go that way.

Done.


> > +      # The TAP tests build an initdb template under build/tmp_install and
> > +      # then `robocopy` it into per-test data directories. Robocopy with the
> > +      # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from
> > +      # their parent dir. On GitHub-hosted Windows runners the workspace's
> > +      # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT
> > +      # grant the runner user (runneradmin) directly. That matters because
> > +      # pg_ctl on Windows uses CreateRestrictedProcess to drop admin
> > +      # privileges from postmaster, so the postmaster process has the user
> > +      # SID in its token but no longer the Administrators group — leaving it
> > +      # with only "Users:(RX)" on pg_control and friends, which causes
> > +      # "PANIC: could not open file global/pg_control: Permission denied".
> > +      #
> > +      # Fix it once on the workspace dir with (OI)(CI) inheritance flags so
> > +      # every file/dir created underneath gets an explicit grant for the
> > +      # current user.
> > +      - name: Grant workspace ACL to runner user
> > +        shell: pwsh
> > +        run: |
> > +          icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null
> > +          Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}"
>
> Perhaps this would be better to fix by changing the robocopy flags?

I couldn't fix this by using robocopy flags. I used /COPYALL and
/SECFIX together but they didn't work.


> > +      # postgres' plpython3u loads python3.dll (the stable-ABI forwarder)
> > +      # which in turn loads whichever python3NN.dll the Windows loader finds
> > +      # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships
> > +      # its own python3.dll + python39.dll and appears on PATH *before* the
> > +      # hostedtoolcache Python 3.12 — so without intervention the backend
> > +      # ends up running Python 3.9 while postgres' stdlib search uses 3.12,
> > +      # producing `ImportError: cannot import name 'text_encoding' from
> > +      # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`).
> > +      #
> > +      # Pin PYTHONHOME to the Python 3.12 prefix, and prepend that prefix
> > +      # to PATH so its python3.dll wins the DLL search.
> > +      - name: Pin Python prefix on PATH and PYTHONHOME
> > +        shell: pwsh
> > +        run: |
> > +          $prefix = (python -c "import sys; print(sys.prefix)").Trim()
> > +          Add-Content $env:GITHUB_ENV "PYTHONHOME=$prefix"
> > +          Add-Content $env:GITHUB_PATH $prefix
> > +          Write-Host "PYTHONHOME=$prefix"
> > +          Write-Host "Prepended $prefix to PATH"
>
> GRJGJKLJKJDFJKDF.

I re-checked this since Jelte wasn't completely sure about this [2]
but this is unfortunately correct :(


> > +      - name: Install dependencies
> > +        shell: pwsh
> > +        run: |
> > +          choco install -y --no-progress --limitoutput diffutils winflexbison
> > +          # meson + ninja aren't preinstalled on windows-2022. Install via pip
> > +          python -m pip install --upgrade meson ninja
> > +
> > +          # OpenSSL 1.1 via the slproweb installer (pinned to match the
> > +          # version used elsewhere in postgres CI).
> > +          curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe
> > +          Start-Process -Wait -FilePath ./openssl-setup.exe `
> > +            -ArgumentList '/DIR=c:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES'
> > +          # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll
> > +          # in c:\openssl\1.1\bin\ and updates the system PATH. GH Actions
> > +          # snapshots PATH at job start though, so the running job won't
> > +          # see those DLLs and initdb.exe would crash silently at runtime.
> > +          # Push the bin dir onto GITHUB_PATH so it persists for later steps.
> > +          Add-Content $env:GITHUB_PATH "c:\openssl\1.1\bin"
>
> I don't like that much, but I'm not sure we have a better alternative
> short-term.

Making chocolatey would be a nice alternative. You already said
sometimes chocolatey takes too much time. I am planning to spend time
on it unless we are planning to use our own Windows containers.


> > +  windows-mingw:
> > +    name: Windows - Server 2022, MinGW64 - Meson
> > +    needs: [setup, sanity-check]
> > +    if: |
> > +      !cancelled() &&
> > +      needs.setup.outputs.mingw == 'true' &&
> > +      needs.sanity-check.result != 'failure'
> > +    runs-on: windows-2022
> > +    timeout-minutes: 60
> > +    env:
> > +      TEST_JOBS: 4  # higher concurrency causes occasional failures
> > +      PG_TEST_USE_UNIX_SOCKETS: 1
> > +      PG_REGRESS_SOCK_DIR: 'c:\pgsock\'
> > +      TAR: "c:/windows/system32/tar.exe"
> > +      # for mingw plpython to find its installation
> > +      PYTHONHOME: D:/a/_temp/msys64/ucrt64
> > +
> > +      MSYS: winjitdebug
> > +      CHERE_INVOKING: 1
> > +      MESON_FEATURES: >-
> > +        -Dnls=disabled
>
> Missing comments from .cirrus.tasks.yml

Done.

v3 is attached. Just a quick note, v3 includes Zsolt [3] And Peter's
[4] reviews & feedback too. I will reply to them after sending this.

GA run after v3 is applied:
https://github.com/nbyavuz/postgres/actions/runs/26587973538


[1]
https://github.com/actions/runner/issues/1182
https://github.com/orgs/community/discussions/185877
[2] https://postgr.es/m/CAGECzQQBCF%3DHSk4eCc1fEYTpCt59rgpcwWp47%2B6M-CDMYEaM2A%40mail.gmail.com
[3] https://postgr.es/m/CAN4CZFO4usEzFQoYzEywvOgoagW%3DU4yhpB4Oq-a7bUCR53djHA%40mail.gmail.com
[4] https://postgr.es/m/3daa29a4-6a08-41c1-8a6a-53ba8cd3c7fb%40eisentraut.org


--
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-05-28T17:07:43Z

Hi,

Thank you for looking into this!

On Thu, 28 May 2026 at 02:10, Zsolt Parragi <zsolt.parragi@percona.com> wrote:
>
> Hello!
>
> I didn't try the workflow on github yet, I only have a few comments
> based on the yaml file:
>
> +on:
> +  push:
> +    branches: [ "*" ]
> +
>
> Should this be "**", otherwise we ignore branches with a "/" in it?

You are right. I completely removed the 'branches:' part, now it
should match with every branch.

> +      SANITIZER_FLAGS: -fsanitize=address
>
> is this used anywhere? Seems like it's directly included into CFLAGS/LDFLAGS.

It wasn't used but I put it there for visibility since in Github
Actions you can't refer to other environment variables (Perhaps a
comment was needed for this). We don't use this now since all linux
tasks are merged in the v3 [1].

[1] https://postgr.es/m/CAN55FZ1-qiOWtQH5o6Q_7LJ7S3Ef_hfDE068uP0hGjB3gzwghg%40mail.gmail.com

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-05-28T17:08:39Z

Hi,

Thank you for looking into this!

On Thu, 28 May 2026 at 14:49, Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 25.05.26 14:14, Nazir Bilal Yavuz wrote:
> > On Tue, 19 May 2026 at 01:27, Nazir Bilal Yavuz<byavuz81@gmail.com> wrote:
> >> I think we can merge these two patches and move forward that way. I am
> >> planning to review your patch and see what I can come up with to get
> >> it to a committable state.
> > Here is the v2, I took Jelte's patch and reviewed & merged it with my
> > patch.
>
> I have tested this patch and inspected the output mostly to make sure
> that there are no regressions about what features and dependency
> versions are being tested (diffed the various logs).  I'm proposing a
> few minor fixups in the attached patch, but other than that (and what
> others have mentioned), this pretty much works well, and I would be
> content to proceed with this or whatever state it's on in a few days.
>
> Some comments in detail:

I addressed these feedbacks in v3 [1].


> - Others have already mentioned about the potential for this to conflict
> with downstream uses of GH Actions.  I suggest renaming the file from
> ci.yml to something like postgresql-ci.yml, so that there is no file
> naming conflict or confusion.

Done.


> - As was already mentioned, the Linux/Meson job is very long (slow) and
> should be split into separate 32/64-bit jobs.
>
> - The job names are too long and get truncated in the UI.  This is
> especially annoying when the important differentiator like "Autoconf" or
> "Meson" gets cut off.  I'm proposing some changes to the job names that
> make them display better.  (Also consider this if you make separate jobs
> for 32/64-bit.  The usable space is about 20 characters.)

I think this is better. Also, I merged all Linux tasks and because of
that I needed to add 64 bit and 32 bit to task names. So, I removed
'Debian' from task names because of the same reason you mentioned.


> - On macOS, there were some dependency differences:
>
>    - readline was not used.
>    - tcl-tk (version 9) was used instead of tcl-tk@8.
>    - python@3.12 was installed but not actually used in the build.
>    - zlib version differed.
>
>    Maybe the zlib difference is not important and could be ignored.
>    Also, maybe we don't need to use a versioned python dependency.  (We
>    didn't have one before we switched Cirrus from Homebrew to MacPorts.)

I used MacPorts like we did in Cirrus. So, I didn't apply these changes.


> - On macOS, the meson setup output reported a significantly different
> sysroot, which was confusing.  I think the sysroot is only used if you
> build against a system perl/python/tcl, which we don't, so I added an
> option to disable the sysroot use.  That way, if we do end up making use
> of the sysroot, someone is forced to investigate this issue.  I don't
> know if this makes sense.

I think it makes sense.


> - For macOS, I threw in some HOMEBREW_* environment variables to disable
> some unnecessary additional output or cleanup steps.

I didn't apply these since MacPorts is used now.


> - On Windows/VS, we should install winflexbison3 not winflexbison, to
> get an up-to-date version.

Done.


> - FUTURE: On Windows/VS, we use openssl 1.1, which matches the Cirrus
> setup, so it's ok, but the equivalent buildfarm members all use openssl
> 3.*, so we should consider upgrading that sometime to make that more
> consistent.

I wondered the same thing while working on this.


> - On Windows/minGW, I dropped a few packages from the set to be
> installed, which didn't seem necessary.

Done.


[1] https://postgr.es/m/CAN55FZ1-qiOWtQH5o6Q_7LJ7S3Ef_hfDE068uP0hGjB3gzwghg%40mail.gmail.com

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Álvaro Herrera <alvherre@kurilemu.de> — 2026-05-28T18:11:45Z

Hi, I just pushed this to github to see how it would behave.  In case
anyone is curious, the run is here,

https://github.com/alvherre/postgres/actions/runs/26591496806

Overall I get the impression that it's much slower than Cirrus.  20
minutes in, only the two "Linux - Meson" build finished, in 14 and 17
minutes respectively.  Macos took 10 minutes just for the macports
install!  Cirrus completed the same build in 18:32,
https://cirrus-ci.com/build/4817054382948352

If I read the Github docs correctly, I get 2000 run minutes for free
each month.  That would mean I can run at most some ... 15 runs per
month?  That sounds quite limiting.  I hope we allow self-hosted runners
at some point; I have quite a bit of spare CPU capacity at home that I
could run the Linux and CompilerWarnings tasks on, leaving the Github
machines to run only the macOS and Windows ones.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
"I'm impressed how quickly you are fixing this obscure issue. I came from 
MS SQL and it would be hard for me to put into words how much of a better job
you all are doing on [PostgreSQL]."
 Steve Midgley, http://archives.postgresql.org/pgsql-sql/2008-08/msg00000.php

Re: Heads Up: cirrus-ci is shutting down June 1st

Álvaro Herrera <alvherre@kurilemu.de> — 2026-05-28T18:42:41Z

On 2026-May-28, Álvaro Herrera wrote:

> Hi, I just pushed this to github to see how it would behave.  In case
> anyone is curious, the run is here,
> 
> https://github.com/alvherre/postgres/actions/runs/26591496806
> 
> Overall I get the impression that it's much slower than Cirrus.  [...]
> 
> If I read the Github docs correctly, I get 2000 run minutes for free
> each month.  That would mean I can run at most some ... 15 runs per
> month?  That sounds quite limiting.

It ended up taking 3 hours 3 minutes, which means I can do 10.9 of those
per month.  Further runs will take less time due to ccache I suppose,
but the actual build is not a huge fraction of the total run time.
Linux+CompilerWarnings totalled 89 minutes; if I subtract those from the
total, I need 94 minutes to run the other builds, and then I can do 21.2
runs per month.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-05-28T19:42:13Z

Hi,

On 2026-05-28 20:42:41 +0200, Álvaro Herrera wrote:
> On 2026-May-28, Álvaro Herrera wrote:
> 
> > Hi, I just pushed this to github to see how it would behave.  In case
> > anyone is curious, the run is here,
> > 
> > https://github.com/alvherre/postgres/actions/runs/26591496806
> > 
> > Overall I get the impression that it's much slower than Cirrus.  [...]

> Macos took 10 minutes just for the macports install!  Cirrus completed the
> same build in 18:32, https://cirrus-ci.com/build/4817054382948352

FWIW, the macports install should be down to ~5s on subsequent runs. Doing
that uncached was also rather slow on cirrus.

> > If I read the Github docs correctly, I get 2000 run minutes for free
> > each month.  That would mean I can run at most some ... 15 runs per
> > month?  That sounds quite limiting.

My understanding is that the 2000 minutes is the limit for *private*
repositories: [1]
"Use of the standard GitHub-hosted runners is free and unlimited on public
repositories."

The concurrency limits are also pretty generous: [2], 20 jobs of 4 cores each.

If you look at the limits of other providers, they are much much lower. Like
400 CPU minutes for gitlab (so about 100 minutes of 4 core VMs).

On 2026-05-28 20:11:45 +0200, Álvaro Herrera wrote:
> Hi, I just pushed this to github to see how it would behave.  In case
> anyone is curious, the run is here,
> 
> https://github.com/alvherre/postgres/actions/runs/26591496806
> 
> Overall I get the impression that it's much slower than Cirrus.  20
> minutes in, only the two "Linux - Meson" build finished, in 14 and 17
> minutes respectively.

It's definitely slower. I've not fully analyzed why, my suspicion is that we
end up being rather terribly IO bound - we used bigger and faster disks on
cirrus than we have access to with github hosted runners (there are large
runners with more storage, but that's not free).

A full testrun on master creates about 36GB of data directories. If individual
tests are fast, that's often not *that* bad, because the tests are over before
linux decides to flush out the data, and then linux never needs to write that
data back, because we remove the data directories immediately. But once you
get to the point that several tests take more than 30s (the default time after
which linux writes dirty data back) or enough dirty data accumulates (20% of
memory IIRC), you have a lot of IO.

My buildfarm host, which hosts quite a few animals, got a new disk within the
last year. Here's what smartctl says about disk IO:

Data Units Read:                    43,513,034 [22.2 TB]
Data Units Written:                 6,062,401,949 [3.10 PB]

A nice indication of how much our tests end up writing...

I think we're going to have to fix that on our end to some degree. 36GB of
data being written each test run is just not reasonable. We spin up quite a
few separate data directories for tests that take well under a second. That's
just a very unfavorable ratio. In [3] I wrote

> I think we need to combine about half the modules in src/test/modules, the
> current course is absurd:
> 
> 16:  37
> 17:  46
> 18:  49
> dev: 62

Greetings,

Andres Freund

[1] https://docs.github.com/en/actions/reference/runners/github-hosted-runners#standard-github-hosted-runners-for-public-repositories
[2] https://docs.github.com/en/actions/reference/limits#job-concurrency-limits-for-github-hosted-runners
[3] https://postgr.es/m/hp4xznm7dqt4ediyhezqysf22eljvu3mucbzsgvgehc6j2hk5v%40laslwlkyixfg

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-05-28T20:50:26Z

On Thu, May 28, 2026 at 10:06 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> > > +          $prefix = (python -c "import sys; print(sys.prefix)").Trim()
> > > +          Add-Content $env:GITHUB_ENV "PYTHONHOME=$prefix"
> > > +          Add-Content $env:GITHUB_PATH $prefix
> > > +          Write-Host "PYTHONHOME=$prefix"
> > > +          Write-Host "Prepended $prefix to PATH"
> >
> > GRJGJKLJKJDFJKDF.
>
> I re-checked this since Jelte wasn't completely sure about this [2]
> but this is unfortunately correct :(

What are the chances we can strip Mercurial out of the PATH instead of
messing with PYTHONHOME? I foresee pain in the future if we override
that globally.

> v3 is attached.

> +        uses: msys2/setup-msys2@v2

Should we pin this? It's the only third-party action we reference, and
Scorecard [1] complains. (I'm not convinced its other complaints in
this category are something we want to worry about, but this caught my
eye.)

We'd need to figure out how to keep it up to date, if we pinned it.
But we probably need to figure out how to keep it up to date anyway.

Scorecard doesn't report any `Dangerous-Workflow` violations, so that's good.

--Jacob

[1] https://scorecard.dev/

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-05-29T09:51:29Z

Hi,

Thank you for looking into this!

On Thu, 28 May 2026 at 23:50, Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
>
> On Thu, May 28, 2026 at 10:06 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> > > > +          $prefix = (python -c "import sys; print(sys.prefix)").Trim()
> > > > +          Add-Content $env:GITHUB_ENV "PYTHONHOME=$prefix"
> > > > +          Add-Content $env:GITHUB_PATH $prefix
> > > > +          Write-Host "PYTHONHOME=$prefix"
> > > > +          Write-Host "Prepended $prefix to PATH"
> > >
> > > GRJGJKLJKJDFJKDF.
> >
> > I re-checked this since Jelte wasn't completely sure about this [2]
> > but this is unfortunately correct :(
>
> What are the chances we can strip Mercurial out of the PATH instead of
> messing with PYTHONHOME? I foresee pain in the future if we override
> that globally.

I think your suggestion is better, done.


> > v3 is attached.
>
> > +        uses: msys2/setup-msys2@v2
>
> Should we pin this? It's the only third-party action we reference, and
> Scorecard [1] complains. (I'm not convinced its other complaints in
> this category are something we want to worry about, but this caught my
> eye.)
>
> We'd need to figure out how to keep it up to date, if we pinned it.
> But we probably need to figure out how to keep it up to date anyway.

Instead of using 'msys2/setup-msys2@v2', I directly installed the
packages by using msys2's package installer (pacman). There was one
problem, it seems that default installation of msys2 is on C: drive
and C: drive is somehow lots slower compared to D: drive [1]. I am
talking about total runtime being ~15 minutes slower (22m -> 35m). So,
I moved msys2 to D: drive and used it from there as a solution.


> Scorecard doesn't report any `Dangerous-Workflow` violations, so that's good.

Nice!

v4 is attached, GA run:
https://github.com/nbyavuz/postgres/actions/runs/26628396798

[1] https://github.com/actions/runner-images/issues/8755


-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-05-29T11:38:17Z

On Fri, May 29, 2026 at 11:51 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
[..]
Hi, thanks to everybody for working on this.

> https://github.com/nbyavuz/postgres/actions/runs/26628396798

Windows (runs-on: windows-2022) seems kind of slow isn't it ?

Maybe that's not related to the patch itself, but any idea why the windows
tests are so slow? Or will we able to somehow accelerate those?

Windows - VS - Meson & ninja / succeeded [..] minutes ago in 31m 28s

Processor(s):              1 Processor(s) Installed.
[..]
Total Physical Memory:     16,379 MB
[..]

but:
NUMBER_OF_PROCESSORS=4
[..]
+      TEST_JOBS: 8

vs

392/396 test_json_parser - postgresql:test_json_parser/002_inline
                 OK              152.56s   3712 subtests passed
393/396 pgbench - postgresql:pgbench/001_pgbench_with_server
                 OK              574.61s   474 subtests passed
394/396 pg_rewind - postgresql:pg_rewind/002_databases
                 OK              772.86s   10 subtests passed
395/396 pg_waldump - postgresql:pg_waldump/001_basic
                 OK              771.19s   156 subtests passed
396/396 libpq_pipeline - postgresql:libpq_pipeline/001_libpq_pipeline
                 OK              395.76s   23 subtests passed

while last CirrusCI run for me for Windows took 19min 21s (4 CPUs / 4 GBs,
but sysinfo reported there "Total Physical Memory: 16,380 MB").

If that's IO traffic as Andres described, maybe we could enable feature
called "Turn off Windows write-cache buffer flushing on the device"
in device manager -> disk -> policies, but dunno how much that would
help really as we seem to be already using fsync=off, maybe it helps
when saving other files too (???)

-J.

Re: Heads Up: cirrus-ci is shutting down June 1st

Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-05-29T13:22:50Z

On Thu, May 28, 2026 at 9:42 PM Andres Freund <andres@anarazel.de> wrote:
[..]
> It's definitely slower. I've not fully analyzed why, my suspicion is that we
> end up being rather terribly IO bound - we used bigger and faster disks on
> cirrus than we have access to with github hosted runners (there are large
> runners with more storage, but that's not free).
>
> A full testrun on master creates about 36GB of data directories. If individual
> tests are fast, that's often not *that* bad, because the tests are over before
> linux decides to flush out the data, and then linux never needs to write that
> data back, because we remove the data directories immediately. But once you
> get to the point that several tests take more than 30s (the default time after
> which linux writes dirty data back) or enough dirty data accumulates (20% of
> memory IIRC), you have a lot of IO.
>
> My buildfarm host, which hosts quite a few animals, got a new disk within the
> last year. Here's what smartctl says about disk IO:
>
> Data Units Read:                    43,513,034 [22.2 TB]
> Data Units Written:                 6,062,401,949 [3.10 PB]
>
> A nice indication of how much our tests end up writing...

`nijna -C build test` (of course without compilation) that was run in
dedicated cgroup gave me this /sys/fs/cgroup/my_test_suite/io.stat
figure:
    rbytes=88616960 wbytes=18406756352 rios=13843 wios=275457 dbytes=0 dios=0

~84.5 MB read
~17.1 GB written (sic!!!)
13k read ops
275k write ops

So yeah, I've really even didn't think we could generate that much IO there.
btw it's seems to be coming from block controller, so it's number of
flushed to disk (so the logically written data but removed without flush?
would be way higher; so by what Your' saying we should tweak that 30s
writeback right?) Anyway I've tried with way more relaxed dirty/writeback and
got this stil onthe same laptop with 32GB RAM:
    rbytes=20934656 wbytes=5957296128 rios=3040 wios=81992 dbytes=0 dios=

~20MB read
~5.55 GB written (down from 17GB)
3k read ops
82k write ops

And that was without even trying hard:
    sudo sysctl -w vm.dirty_ratio=50 # ~16GB
    sudo sysctl -w vm.dirty_background_ratio=40
    sudo sysctl -w vm.dirty_expire_centisecs=60000 #default was 3000 as You said
    sudo sysctl -w vm.dirty_writeback_centisecs=50000

Steps were:
  sudo mkdir /sys/fs/cgroup/my_test_suite
  echo $$ | sudo tee /sys/fs/cgroup/my_test_suite/cgroup.procs
  ninja -C build test
  cat /sys/fs/cgroup/my_test_suite/io.stat

-J.

Re: Heads Up: cirrus-ci is shutting down June 1st

Peter Eisentraut <peter@eisentraut.org> — 2026-05-29T15:47:06Z

On 28.05.26 19:06, Nazir Bilal Yavuz wrote:
> I found we can use matrices and merged all linux tasks. I am not sure
> that is better since it is a bit harder to read now.

Yeah, I think the way it looks now (with the matrix) is quite confusing.

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-05-29T15:56:19Z

Hi,

On 2026-05-29 13:38:17 +0200, Jakub Wartak wrote:
> On Fri, May 29, 2026 at 11:51 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> [..]
> Hi, thanks to everybody for working on this.
> 
> > https://github.com/nbyavuz/postgres/actions/runs/26628396798
> 
> Windows (runs-on: windows-2022) seems kind of slow isn't it ?
> 
> Maybe that's not related to the patch itself, but any idea why the windows
> tests are so slow? Or will we able to somehow accelerate those?
> 
> Windows - VS - Meson & ninja / succeeded [..] minutes ago in 31m 28s
> 
> Processor(s):              1 Processor(s) Installed.
> [..]
> Total Physical Memory:     16,379 MB
> [..]
> 
> but:
> NUMBER_OF_PROCESSORS=4
> [..]
> +      TEST_JOBS: 8
> 
> vs
> 
> 392/396 test_json_parser - postgresql:test_json_parser/002_inline
>                  OK              152.56s   3712 subtests passed
> 393/396 pgbench - postgresql:pgbench/001_pgbench_with_server
>                  OK              574.61s   474 subtests passed
> 394/396 pg_rewind - postgresql:pg_rewind/002_databases
>                  OK              772.86s   10 subtests passed
> 395/396 pg_waldump - postgresql:pg_waldump/001_basic
>                  OK              771.19s   156 subtests passed
> 396/396 libpq_pipeline - postgresql:libpq_pipeline/001_libpq_pipeline
>                  OK              395.76s   23 subtests passed
> 
> while last CirrusCI run for me for Windows took 19min 21s (4 CPUs / 4 GBs,
> but sysinfo reported there "Total Physical Memory: 16,380 MB").

The difference here likely is due to the different type of CPU cores. On
cirrus, we got 4 non-SMT cores (because the type of CPU used didn't use SMT),
whereas on GHA we have 4 hardware threads, but only two real cores.


> If that's IO traffic as Andres described, maybe we could enable feature
> called "Turn off Windows write-cache buffer flushing on the device"
> in device manager -> disk -> policies, but dunno how much that would
> help really as we seem to be already using fsync=off, maybe it helps
> when saving other files too (???)

I think I was wrong about IO being the main issue. I've measured the CPU
utilization during a linux run, and basically it's 100% busy during the whole
test run (baring the first and last few seconds).  Which does seem to mainly
point to the difference being simply that we just have half the real cores as
we had before.

I do see higher %sys CPU utilization than I'd expect, so that may be worth
investigating.

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Peter Eisentraut <peter@eisentraut.org> — 2026-05-30T20:51:16Z

On 29.05.26 11:51, Nazir Bilal Yavuz wrote:
> v4 is attached, GA run:
> https://github.com/nbyavuz/postgres/actions/runs/26628396798

This looks fine to me.

Attached are a few more small tweaks.

If you produce another patch, maybe you could expand the commit message 
and add all the credits, the help the eventual committer.

(Also note that there is a freeze on the master branch until the beta is 
tagged, so this likely wouldn't be committable until late Monday or 
Tuesday, so there is still some time to discuss and fine-tune.)

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-05-31T01:42:35Z

Hi,

On 2026-05-29 12:51:29 +0300, Nazir Bilal Yavuz wrote:
> +      - name: Upload logs
> +        if: failure()
> +        uses: actions/upload-artifact@v7
> +        with:
> +          name: windows-vs-logs-${{ github.run_id }}
> +          path: |
> +            ${{ env._LOG_PATHS }}
> +            crashlog-*.txt
> +          if-no-files-found: ignore

You're collecting crash logs here, but that doesn't ever work, because we only
got them thanks to the windows image setting things up that way
(c.f. scripts/windows_install_dbg.ps1).

Looks like cdb.exe is actually installed in the windows runner images, so we
just need to use the registry settings from that file.

I'm working on a bunch of other edits (planning to post them soon, need a bit
more work), but I am currently not planning to look at that aspect
immediately.

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-05-31T06:39:23Z

Hi,

On Fri, 29 May 2026 at 18:47, Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 28.05.26 19:06, Nazir Bilal Yavuz wrote:
> > I found we can use matrices and merged all linux tasks. I am not sure
> > that is better since it is a bit harder to read now.
>
> Yeah, I think the way it looks now (with the matrix) is quite confusing.

I agree. I still can't decide which one is better, so I am okay with either one.


--
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-05-31T06:43:03Z

Hi,

On Sat, 30 May 2026 at 23:51, Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 29.05.26 11:51, Nazir Bilal Yavuz wrote:
> > v4 is attached, GA run:
> > https://github.com/nbyavuz/postgres/actions/runs/26628396798
>
> This looks fine to me.
>
> Attached are a few more small tweaks.

Thank you!

I applied all of them with one small change, I added a comment
explaining the 'MINGW_PACKAGE_PREFIX' environment variable.

> If you produce another patch, maybe you could expand the commit message
> and add all the credits, the help the eventual committer.

Done.

v5 is attached. GA run:
https://github.com/nbyavuz/postgres/actions/runs/26705141176

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-05-31T06:44:09Z

Hi,

On Sun, 31 May 2026 at 04:42, Andres Freund <andres@anarazel.de> wrote:
>
> On 2026-05-29 12:51:29 +0300, Nazir Bilal Yavuz wrote:
> > +      - name: Upload logs
> > +        if: failure()
> > +        uses: actions/upload-artifact@v7
> > +        with:
> > +          name: windows-vs-logs-${{ github.run_id }}
> > +          path: |
> > +            ${{ env._LOG_PATHS }}
> > +            crashlog-*.txt
> > +          if-no-files-found: ignore
>
> You're collecting crash logs here, but that doesn't ever work, because we only
> got them thanks to the windows image setting things up that way
> (c.f. scripts/windows_install_dbg.ps1).
>
> Looks like cdb.exe is actually installed in the windows runner images, so we
> just need to use the registry settings from that file.

Ah, I see. I removed crashlog-*.txt for now and added a comment to v5
that it needs to be fixed [1]. Command in the
'scripts/windows_install_dbg.ps1' is more complicated than I thought
[2], I need to work on it.

[1] https://postgr.es/m/CAN55FZ3kE%2B_yM0akRAXBWM2rLnprEROB%2BJSEPrp1mvchdMBYBg%40mail.gmail.com
[2] https://github.com/anarazel/pg-vm-images/blob/main/scripts/windows_install_dbg.ps1#L56C1-L56C229

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-01T10:01:58Z

Hi Andres,

On Fri, May 29, 2026 at 5:56 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2026-05-29 13:38:17 +0200, Jakub Wartak wrote:
> > On Fri, May 29, 2026 at 11:51 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> > [..]
> > Hi, thanks to everybody for working on this.
> >
> > > https://github.com/nbyavuz/postgres/actions/runs/26628396798
> >
> > Windows (runs-on: windows-2022) seems kind of slow isn't it ?
> >
> > Maybe that's not related to the patch itself, but any idea why the windows
> > tests are so slow? Or will we able to somehow accelerate those?
> >
> > Windows - VS - Meson & ninja / succeeded [..] minutes ago in 31m 28s
> >
> > Processor(s):              1 Processor(s) Installed.
> > [..]
> > Total Physical Memory:     16,379 MB
> > [..]
> >
> > but:
> > NUMBER_OF_PROCESSORS=4
> > [..]
> > +      TEST_JOBS: 8
> >
> > vs
> >
> > 392/396 test_json_parser - postgresql:test_json_parser/002_inline
> >                  OK              152.56s   3712 subtests passed
> > 393/396 pgbench - postgresql:pgbench/001_pgbench_with_server
> >                  OK              574.61s   474 subtests passed
> > 394/396 pg_rewind - postgresql:pg_rewind/002_databases
> >                  OK              772.86s   10 subtests passed
> > 395/396 pg_waldump - postgresql:pg_waldump/001_basic
> >                  OK              771.19s   156 subtests passed
> > 396/396 libpq_pipeline - postgresql:libpq_pipeline/001_libpq_pipeline
> >                  OK              395.76s   23 subtests passed
> >
> > while last CirrusCI run for me for Windows took 19min 21s (4 CPUs / 4 GBs,
> > but sysinfo reported there "Total Physical Memory: 16,380 MB").
>
> The difference here likely is due to the different type of CPU cores. On
> cirrus, we got 4 non-SMT cores (because the type of CPU used didn't use SMT),
> whereas on GHA we have 4 hardware threads, but only two real cores.
>
>
> > If that's IO traffic as Andres described, maybe we could enable feature
> > called "Turn off Windows write-cache buffer flushing on the device"
> > in device manager -> disk -> policies, but dunno how much that would
> > help really as we seem to be already using fsync=off, maybe it helps
> > when saving other files too (???)
>
> I think I was wrong about IO being the main issue. I've measured the CPU
> utilization during a linux run, and basically it's 100% busy during the whole
> test run (baring the first and last few seconds).  Which does seem to mainly
> point to the difference being simply that we just have half the real cores as
> we had before.
>
> I do see higher %sys CPU utilization than I'd expect, so that may be worth
> investigating.

So I've spent half of day on trying to see what makes the tests so slow at
least in my case. I can also confirm %CPU combined (with high 33% sys).

0. baseline was ~71s (stuff already hot)
1a. down to 64s with dirtywriteback tune (and mostly to avoid NVMe/SSD wear)
1b. ~65s with tmpfs, so I've left using dirtywriteback sysctls:
    sudo mount -t tmpfs -o size=4G,uid=XXX,mode=755  tmpfs build/tmp_install
    sudo mount -t tmpfs -o size=16G,uid=XXX,mode=755 tmpfs /build/testrun
2. Splitting the tests (isolation, 027_stream_regress, pg_upgrade) into 4
   parallel streams of each did not help much (they are longest ones)
3. I've spotted the falcon-sensor (EDR agent, using eBPF) very busy, so
   I've shut it down, got the duratiion down to 43s.
4. Still for that 43s dominant factor was the mmap/page-fault/PTEs related
   to the number of backends we spawn. Literally later when I put
   Claude  to work he said to me this "Backend startup costs roughly 2.5x
   as much as the actual queries". And later when I've pushed to count using
   log_connections it said "Got 24,903 total connections in 46 s = 541
   backend forks/second." and got this top report:
     8,610   subscription      - 35 % of all connections in the suite
     4,382   recovery          - 18 %
     1,100   pg_upgrade
       896   isolation
       694   pg_dump
       682   pg_basebackup

    Fixing above subscription to ~5000 conns did not gain much (well it saved
    5% of runtime 43s -> 41s). It's literally 10k lines of
    s/$node_subscriber->safe_psql/sub_bg->query_safe/g across dozens of files
    in src/test/subscription/t/). Too big for review and I'm not sharing as
    it could contain errors.

5. Spotted that we do plenty of initdb and cached-initdb (cp), so I had idea
   about XFS's cp reflinks=always in build/, but I couldn't do that without
   /dev/loop, so apparently XFS (reflink=1) vs ext4(reflink=0) halves number
   of writes while even still on /dev/loop device, but that somehow
   does not directly contribute to duration of the test (well we are
   bottlenecked on CPU anyway, so this is just smarter? way of avoiding I/O;
   maybe with cold-caches and on real VMs running with XFS would be faster)

   +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
   @@ -687,7 +687,13 @@ sub init
                  }
                  else
                  {
   -                       @copycmd = qw(cp -RPp);
   +                       @copycmd = qw(cp --reflink=always -RPp);


Other interesting ideas: pg_regress with built-in connection pool (IMHO not
worth it), mitigations=off (to avoid syscalls being taxed, got not
improvement with this).

As for the Windows, I don't have better idea than the just avoid I/O if possible
("Turn off Windows write-cache buffer flushing on the device"), sorry(!), and
maybe throwing in more bigger box... ;]

-J.

Re: Heads Up: cirrus-ci is shutting down June 1st

Álvaro Herrera <alvherre@kurilemu.de> — 2026-06-01T14:41:35Z

On 2026-Jun-01, Jakub Wartak wrote:

> 5. Spotted that we do plenty of initdb and cached-initdb (cp), so I had idea
>    about XFS's cp reflinks=always in build/, but I couldn't do that without
>    /dev/loop, so apparently XFS (reflink=1) vs ext4(reflink=0) halves number
>    of writes while even still on /dev/loop device, but that somehow
>    does not directly contribute to duration of the test (well we are
>    bottlenecked on CPU anyway, so this is just smarter? way of avoiding I/O;
>    maybe with cold-caches and on real VMs running with XFS would be faster)

I wonder if we could somehow make several test use the same instance
instead of each creating its own.  Look at src/bin/scripts/t/ for
instance: we have 14 Cluster->init() calls there, 12 of which have no
arguments; that means we could use just one instance for those 12 and it
would mostly be fine.  (There is some degree of dependency: for example
if two tests create the same database, that would at present work fine,
but with a common instance it'd fail.  Should be easy to fix though.)

The only real problem is that we have no mechanism to share the same
instance across several .pl files.  Maybe there's a way to do this in a
smarter way?

Maybe have 000_create.pl that does nothing other than Cluster->new(...)
and Cluster->init(), and then the other scripts simply do
->init_from_environment() to reuse that instance.  Or something like
that.

-- 
Álvaro Herrera               48°01'N 7°57'E  —  https://www.EnterpriseDB.com/
"Porque Kim no hacía nada, pero, eso sí,
con extraordinario éxito" ("Kim", Kipling)

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-01T21:57:15Z

Hi,

Attached is an large incremental patch onto Bilal's version:

- I hacked enough on the pg-vm-images repo to make it store containers in
  github (on the gha_main branch, for now).  That makes the container faster
  and cheaper to retrieve.

  We might want to split the containers further (e.g. cross building and 32bit
  support), but for now I just split the docs stuff into a separate container.


- The reason that sometimes cancelling took a long time was, afaict, the use
  of always() in the compiler-warning job. That apparently prevents GHA from
  cancelling the job in a timely fashion. I turned those into !cancelled().

  Also addded a !cancelled() to SanityCheck.


- In the end I didn't like the matrix all that much, makes it considerably
  harder to understand everything and the restrictions GHA puts on it are just
  too annoying.

  I replaced it much more heavy use of yaml anchors/aliases and by updating
  the environment programattically.  I'm not sure how much better it is now.


- The macports cache keys included the run_id, I think that's a bad idea,
  because it leads to the cache being newly uploaded even if there's been no
  change. That'll lead to even more quickly churning through the cache space.


- ccache:

  - There was too much duplication around the ccache handling for my taste. I
    moved that into a yaml anchor and reused it everywhere.  By using
    ${{github.job_id}} the cache names don't need to be manually disambiguated.

  - The ccache names weren't unique enough. The run_id doesn't change during
    reruns which would lead to warnings.

  - Made it so that the cache is saved immediately after the build, so that
    cancelled builds still save the cache.


- I wanted to share commands like meson test between the tasks, made that work
  with a bit of hackery.


- I didn't see why
    find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \;
  was needed.


- I found it hard to find actual warnings in the output of CompilerWarnings,
  due to to all the configure output. I added some magic output stuff to make
  the configure output collapse once it ran.


- I converted most things to use ${{env.varname}} instead of $VARNAME or
  %VARNAME%, as that allows sharing code between windows and other OSs if one
  is a bit careful.


- Deduplicated a few other things like logging.


- Added a commit removing cirrus, mainly because I was tired of it also
  running while hacking on this.


- Also added a commit to reduce the segment size in tests to 1MB, that makes
  them a decent bit less IO intensive.


- Added a commit to just run regress/regress, makes it a lot faster to test
  "complete-ish" cycles.


- Changed sanitizer using builds to use -O2, to reduce the CPU cost a bit.

  This makes uncached builds noticeably slower, but does appear to be a
  win. But I could see counter-arguments to that too.


- Removed :detect_stack_use_after_return=0, as that's been made unnecessary
  since you "forked off" from .cirrus.tasks.yml.


A few other comments:

- All the caches in GHA apparently are branch specific, except that branches
  can access the caches of the main branch.

  I think that'll make particularly macports very expensive for cfbot.  I
  wonder if we ought to build the "base" macports cache in pg-vm-images.

  Similarly, I think we probably should do that for mingw64.


- src/tools/ci/README needs updating


- I experimented with making the slowest jobs faster by using
    meson test --slice 1/2
  that actually does nicely work. But it didn't quite seem critical for now.


Here's a run running with just regress/regress:
  https://github.com/anarazel/postgres/actions/runs/26782558353
(the CompilerWarnings ccache hit had a miss, that's why it's not faster)

Here's a (not yet completed) run with the full tests:
  https://github.com/anarazel/postgres/actions/runs/26784248887


Thoughts?


Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-02T10:13:09Z

Hi Andres/Nazir,

On Mon, Jun 1, 2026 at 11:57 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> Attached is an large incremental patch onto Bilal's version:
[..]

> Thoughts?

The timings I got during morning EMEA hours [1] (my first GHA run) was
with all those commits except last one, and it's LGTM (it's a flawless
experience
at this point as end user; all succeed). Perhaps my only worry would be
success of this and proliferation of using this often, where plenty of people
hit download those packages at runtime (MacOS/Win) way too often (or
download sites being unavailable and causing outages in the pipeline).

Continuing on previous story...:
Windows was still @ 31mins, and whatever I've tried it is was not helping it
(but I cannot measure inside GHA Runner what was happening, so those were blind
shots with fstweaks, etc). One important thing, altough I failed altering
CacheIsPowerProtected (avoid flushing the write cache) as it seems impossible
for me to do so on D:\ (as paging file is there and and altering it also
requires reboot), at least we know stuff is way slower than it could be on
those runners:

"Get-PhysicalDisk | Get-StorageAdvancedProperty" reported:

FriendlyName      SerialNumber IsPowerProtected IsDeviceCacheEnabled
------------      ------------ ---------------- --------------------
Msft Virtual Disk                         False                False
Msft Virtual Disk                         False                False

Perhaps there's way to use some custom image/templ with different settings,
especially for D:\, after all it's just volatile stuff. Thoughts? (not that I
care that much for Win, but waiting half hour for it finish every time is
not going to be nice...)

Also, maybe that's not useful for for GHA/CI, but for folks trying to local test
on win32:

I've run some quick test on my own Windows VM with manual "ninja test".
Immediatley spotted Defener as top#1 CPU. In GHA workflow we already
this thing for disabling Defender AV:
    "Set-MpPreference -DisableRealtimeMonitoring $true
        -SubmitSamplesConsent NeverSend -MAPSReporting Disable"

Yet, when I re-run the verification command it showed still as as enabled (!)
and I've still got Defender as top#1 CPU during next ninja run, so disabling
RealTimeMonitoring didn't kick in? So I went and I've manually disabled
"real-time protection" in "Virus & thread protection settings" in the control
panel as it was still on after above and only then it got a nice boost
(even visually when watching tests). Gemini told me later that there's
"TamperProtection" thing (!) to ignore powershell. see e.g.
    "Get-MpComputerStatus | Select-Object IsTamperProtected"
and it requires manual human steps to disarm... but apparently all of this is
not necessary on GHA...

-J.

[1] - https://github.com/jakubwartakEDB/postgres-public-ci/actions/runs/26806653438

Re: Heads Up: cirrus-ci is shutting down June 1st

Thomas Munro <thomas.munro@gmail.com> — 2026-06-02T11:59:35Z

On Wed, May 27, 2026 at 3:54 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Tue, May 19, 2026 at 9:22 AM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
> > 3. There's currently no integration with the CFBot or commitfest yet. @Thomas
>
> Looking into this part urgently...

https://commitfest.postgresql.org/patch/6785/ (ie this thread only
since the GHA patches aren't in master yet) is now showing results
that travelled through the new pipeline.  Obviously the URLs when you
click them need some adjustment.  It will take a bit more work to get
the (little used?) log/test analysis stuff back up and running.
Hopefully we can get some of that sorted out in the next day or so...

The CI patches seem to work very nicely so far.

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-06-02T12:19:02Z

Hi,

On Tue, 2 Jun 2026 at 00:57, Andres Freund <andres@anarazel.de> wrote:
>
> Attached is an large incremental patch onto Bilal's version:

Thank you!


> - I hacked enough on the pg-vm-images repo to make it store containers in
>   github (on the gha_main branch, for now).  That makes the container
faster
>   and cheaper to retrieve.

Nice.


>   We might want to split the containers further (e.g. cross building and
32bit
>   support), but for now I just split the docs stuff into a separate
container.

Is this really useful? I remember that we merged Windows VM images because
of the storage costs.


> - The reason that sometimes cancelling took a long time was, afaict, the
use
>   of always() in the compiler-warning job. That apparently prevents GHA
from
>   cancelling the job in a timely fashion. I turned those into
!cancelled().
>
>   Also addded a !cancelled() to SanityCheck.

That makes sense.


> - In the end I didn't like the matrix all that much, makes it considerably
>   harder to understand everything and the restrictions GHA puts on it are
just
>   too annoying.
>
>   I replaced it much more heavy use of yaml anchors/aliases and by
updating
>   the environment programattically.  I'm not sure how much better it is
now.

I think this is better compared to matrix and closer to what we had with
Cirrus. One thing I didn't like is that we need to define yaml anchors in
the jobs, I wish we could define them at the top..


> - The macports cache keys included the run_id, I think that's a bad idea,
>   because it leads to the cache being newly uploaded even if there's been
no
>   change. That'll lead to even more quickly churning through the cache
space.

That makes sense.


> - ccache:
>
>   - There was too much duplication around the ccache handling for my
taste. I
>     moved that into a yaml anchor and reused it everywhere.  By using
>     ${{github.job_id}} the cache names don't need to be manually
disambiguated.
>
>   - The ccache names weren't unique enough. The run_id doesn't change
during
>     reruns which would lead to warnings.
>
>   - Made it so that the cache is saved immediately after the build, so
that
>     cancelled builds still save the cache.

AFAIU, we save cache although the job might fail later. Would that cause
problems?


> - I wanted to share commands like meson test between the tasks, made that
work
>   with a bit of hackery.
>
>
> - I didn't see why
>     find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \;
>   was needed.

I think the reasoning is that we don't have a 'setup_core_files_script'
step on the Cirrus' SanityCheck. I agree we don't need this now.


> - I found it hard to find actual warnings in the output of
CompilerWarnings,
>   due to to all the configure output. I added some magic output stuff to
make
>   the configure output collapse once it ran.

Adding a comment explaining what these echos for would be useful but I
couldn't find a nice place to add a comment. Perhaps the first use of these
echos?


> - I converted most things to use ${{env.varname}} instead of $VARNAME or
>   %VARNAME%, as that allows sharing code between windows and other OSs if
one
>   is a bit careful.

Nice.


> - Deduplicated a few other things like logging.
>
>
> - Added a commit removing cirrus, mainly because I was tired of it also
>   running while hacking on this.

I am not sure that is the actual proposed commit for removing Cirrus but we
have several more places which mention Cirrus.


> - Also added a commit to reduce the segment size in tests to 1MB, that
makes
>   them a decent bit less IO intensive.
>
>
> - Added a commit to just run regress/regress, makes it a lot faster to
test
>   "complete-ish" cycles.

I think that would be useful. Perhaps we can enable this with commit
messages like how 'ci-os-only' is used. I mean something like
'ci-test-only:' or such?


> - Changed sanitizer using builds to use -O2, to reduce the CPU cost a bit.
>
>   This makes uncached builds noticeably slower, but does appear to be a
>   win. But I could see counter-arguments to that too.
>
>
> - Removed :detect_stack_use_after_return=0, as that's been made
unnecessary
>   since you "forked off" from .cirrus.tasks.yml.
>
>
> A few other comments:
>
> - All the caches in GHA apparently are branch specific, except that
branches
>   can access the caches of the main branch.
>
>   I think that'll make particularly macports very expensive for cfbot.  I
>   wonder if we ought to build the "base" macports cache in pg-vm-images.
>
>   Similarly, I think we probably should do that for mingw64.

If i understood correctly, you mean that we will push cache to hardcoded
URL and download caches from the same URL, right? Or do you have something
else in your mind?


> - src/tools/ci/README needs updating

Yes, with several other files:
- src/bin/pg_combinebackup/t/010_hardlink.pl
- src/test/perl/PostgreSQL/Test/Cluster.pm
- src/tools/ci/gcp_ram_disk.sh (Do we need to remove this? I am not sure we
will use this when we have BSDs on GHA.)


I looked at 0002 for now and it looks good to me. I implemented several
more changes on top of 0002 as 0003 and reattached patches:

- Fixed the cancelling on REL_ branches.
- Used YAML anchors on the CompilerWarnings task.


-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-02T15:40:40Z

Hi,

On 2026-06-02 15:19:02 +0300, Nazir Bilal Yavuz wrote:
> On Tue, 2 Jun 2026 at 00:57, Andres Freund <andres@anarazel.de> wrote:
> >   We might want to split the containers further (e.g. cross building and
> 32bit
> >   support), but for now I just split the docs stuff into a separate
> container.
>
> Is this really useful? I remember that we merged Windows VM images because
> of the storage costs.

It doesn't really add storage cost here, because this is containers, where the
different containers are layered ontop of each other. That's not the case for
images.  With images there also was just no gain, because they don't have to
be pulled fully ahead of time, they were (at least at some point, I think not
initially when we started using them) populated on demand.


> > - In the end I didn't like the matrix all that much, makes it considerably
> >   harder to understand everything and the restrictions GHA puts on it are
> just
> >   too annoying.
> >
> >   I replaced it much more heavy use of yaml anchors/aliases and by
> updating
> >   the environment programattically.  I'm not sure how much better it is
> now.
>
> I think this is better compared to matrix and closer to what we had with
> Cirrus. One thing I didn't like is that we need to define yaml anchors in
> the jobs, I wish we could define them at the top..

Agreed...


> >   - There was too much duplication around the ccache handling for my
> taste. I
> >     moved that into a yaml anchor and reused it everywhere.  By using
> >     ${{github.job_id}} the cache names don't need to be manually
> disambiguated.
> >
> >   - The ccache names weren't unique enough. The run_id doesn't change
> during
> >     reruns which would lead to warnings.
> >
> >   - Made it so that the cache is saved immediately after the build, so
> that
> >     cancelled builds still save the cache.
>
> AFAIU, we save cache although the job might fail later. Would that cause
> problems?

I think we explicitly *want* that, as otherwise a job repeatedly failing
causes each attempt to start with a cold cache. Making iterating on the work
more painful.


> > - Deduplicated a few other things like logging.
> >
> >
> > - Added a commit removing cirrus, mainly because I was tired of it also
> >   running while hacking on this.
>
> I am not sure that is the actual proposed commit for removing Cirrus but we
> have several more places which mention Cirrus.

It was not yet.


> > - Also added a commit to reduce the segment size in tests to 1MB, that
> makes
> >   them a decent bit less IO intensive.
> >
> >
> > - Added a commit to just run regress/regress, makes it a lot faster to
> test
> >   "complete-ish" cycles.
>
> I think that would be useful. Perhaps we can enable this with commit
> messages like how 'ci-os-only' is used. I mean something like
> 'ci-test-only:' or such?

The problem is that we have two different ways of phrasing this, for autoconf
and meson...


> > - Changed sanitizer using builds to use -O2, to reduce the CPU cost a bit.
> >
> >   This makes uncached builds noticeably slower, but does appear to be a
> >   win. But I could see counter-arguments to that too.
> >
> >
> > - Removed :detect_stack_use_after_return=0, as that's been made
> unnecessary
> >   since you "forked off" from .cirrus.tasks.yml.
> >
> >
> > A few other comments:
> >
> > - All the caches in GHA apparently are branch specific, except that
> branches
> >   can access the caches of the main branch.
> >
> >   I think that'll make particularly macports very expensive for cfbot.  I
> >   wonder if we ought to build the "base" macports cache in pg-vm-images.
> >
> >   Similarly, I think we probably should do that for mingw64.
>
> If i understood correctly, you mean that we will push cache to hardcoded
> URL and download caches from the same URL, right? Or do you have something
> else in your mind?

I think we'd have to move the caching into pg-vm-images or such for it to
work...


> I looked at 0002 for now and it looks good to me. I implemented several
> more changes on top of 0002 as 0003 and reattached patches:
>
> - Fixed the cancelling on REL_ branches.

Thanks.  I think maybe we should add an && endsWith(..., '_STABLE'), so one
can have WIP progress for stable branches without triggering this (e.g. for
hacking on backports).

> - Used YAML anchors on the CompilerWarnings task.

Nice!

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-02T15:57:34Z

Hi,

On 2026-05-28 20:08:39 +0300, Nazir Bilal Yavuz wrote:
> > - Others have already mentioned about the potential for this to conflict
> > with downstream uses of GH Actions.  I suggest renaming the file from
> > ci.yml to something like postgresql-ci.yml, so that there is no file
> > naming conflict or confusion.
> 
> Done.

I find postgresql-ci.yml a bit long, how about postgres-ci.yml or pg-ci.yml?

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-02T16:53:02Z

Hi,

On 2026-06-02 15:19:02 +0300, Nazir Bilal Yavuz wrote:
> +concurrency:
> +  group: ${{ github.workflow }}-${{ github.ref }}
> +  # Never cancel in-progress runs on master to ensure all commits are tested.
> +  cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}

>  concurrency:
>    group: ${{ github.workflow }}-${{ github.ref }}
> -  # Never cancel in-progress runs on master to ensure all commits are tested.
> -  # FIXME: Should also not cancel REL_XY_STABLE
> -  cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
> +  # Never cancel in-progress runs on master or release branches, to ensure
> +  # all commits are tested.
> +  cancel-in-progress: ${{ github.ref != 'refs/heads/master' && !startsWith(github.ref, 'refs/heads/REL_') }}

Experimenting with this, it actually seems neither really does what I think we
want.

What I think we want is cirrus' behaviour, that is, not cancel workflows that
are on master/stable branches and, if on such a branch, allow multiple runs to
be in progress at the same time.

The above achieves not cancelling workflows on those branches, but it does so
by waiting for the prior workflow to finish.  Which I think is kinda bad,
because it makes it entirely possible to fall further and further behind.

As-is, it'd afaik still cancel workflows, as soon as more than one workflow is
queued (we'd have to set queue: max to allow that).

Afaict what we should do is to instead set a unique group: when on a stable
branch.  Something like

concurrency:
  # For anything other than stable branches, we want there to only be one
  # workflow active for that branch. But on stable branches & master, we
  # neither want to wait for prior runs, nor to cancel them, so that each
  # separately pushed commit is tested.  We achieve that by setting a unique
  # concurrency group when on such a banch.
  group: |
    ${{github.workflow }}-${{
    case(github.ref == 'refs/heads/master' ||
         (startsWith(github.ref, 'refs/heads/REL_') && endsWith(github.ref, '_STABLE')),
         github.run_id,
         github.ref)
    }}

seems to do the trick.

Does that make sense as an approach and goal?

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-06-02T17:36:08Z

Hi,

On Tue, 2 Jun 2026 at 19:53, Andres Freund <andres@anarazel.de> wrote:
>
> On 2026-06-02 15:19:02 +0300, Nazir Bilal Yavuz wrote:
> > +concurrency:
> > +  group: ${{ github.workflow }}-${{ github.ref }}
> > +  # Never cancel in-progress runs on master to ensure all commits are tested.
> > +  cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
>
> >  concurrency:
> >    group: ${{ github.workflow }}-${{ github.ref }}
> > -  # Never cancel in-progress runs on master to ensure all commits are tested.
> > -  # FIXME: Should also not cancel REL_XY_STABLE
> > -  cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
> > +  # Never cancel in-progress runs on master or release branches, to ensure
> > +  # all commits are tested.
> > +  cancel-in-progress: ${{ github.ref != 'refs/heads/master' && !startsWith(github.ref, 'refs/heads/REL_') }}
>
> Experimenting with this, it actually seems neither really does what I think we
> want.
>
> What I think we want is cirrus' behaviour, that is, not cancel workflows that
> are on master/stable branches and, if on such a branch, allow multiple runs to
> be in progress at the same time.
>
> The above achieves not cancelling workflows on those branches, but it does so
> by waiting for the prior workflow to finish.  Which I think is kinda bad,
> because it makes it entirely possible to fall further and further behind.
>
> As-is, it'd afaik still cancel workflows, as soon as more than one workflow is
> queued (we'd have to set queue: max to allow that).
>
>
> Afaict what we should do is to instead set a unique group: when on a stable
> branch.  Something like
>
> concurrency:
>   # For anything other than stable branches, we want there to only be one
>   # workflow active for that branch. But on stable branches & master, we
>   # neither want to wait for prior runs, nor to cancel them, so that each
>   # separately pushed commit is tested.  We achieve that by setting a unique
>   # concurrency group when on such a banch.
>   group: |
>     ${{github.workflow }}-${{
>     case(github.ref == 'refs/heads/master' ||
>          (startsWith(github.ref, 'refs/heads/REL_') && endsWith(github.ref, '_STABLE')),
>          github.run_id,
>          github.ref)
>     }}
>
> seems to do the trick.
>
>
> Does that make sense as an approach and goal?

You are right, I confirm that this approach solves the problem and I
think this solution makes sense.

From another thread:

On Tue, 2 Jun 2026 at 18:57, Andres Freund <andres@anarazel.de> wrote:
>
> On 2026-05-28 20:08:39 +0300, Nazir Bilal Yavuz wrote:
> > > - Others have already mentioned about the potential for this to conflict
> > > with downstream uses of GH Actions.  I suggest renaming the file from
> > > ci.yml to something like postgresql-ci.yml, so that there is no file
> > > naming conflict or confusion.
> >
> > Done.
>
> I find postgresql-ci.yml a bit long, how about postgres-ci.yml or pg-ci.yml?

I am okay with both but my choice would be postgres-ci.yml.

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Peter Eisentraut <peter@eisentraut.org> — 2026-06-02T18:08:17Z

On 01.06.26 23:57, Andres Freund wrote:
> Attached is an large incremental patch onto Bilal's version:

This looks good to me.

(I suppose the 0005 patch was just for testing.  The 0004 patch looks ok 
but could use some explanation in both the commit message and the code.)

+  # FIXME: Should we also run on PRs?

I don't know why we would, since we don't use them.

Re: Heads Up: cirrus-ci is shutting down June 1st

Peter Eisentraut <peter@eisentraut.org> — 2026-06-02T18:08:53Z

On 02.06.26 17:57, Andres Freund wrote:
> Hi,
> 
> On 2026-05-28 20:08:39 +0300, Nazir Bilal Yavuz wrote:
>>> - Others have already mentioned about the potential for this to conflict
>>> with downstream uses of GH Actions.  I suggest renaming the file from
>>> ci.yml to something like postgresql-ci.yml, so that there is no file
>>> naming conflict or confusion.
>>
>> Done.
> 
> I find postgresql-ci.yml a bit long, how about postgres-ci.yml or pg-ci.yml?

If we're worried about length, then let's use the latter.

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-02T18:38:37Z

Hi,

On 2026-06-01 12:01:58 +0200, Jakub Wartak wrote:
> So I've spent half of day on trying to see what makes the tests so slow at
> least in my case. I can also confirm %CPU combined (with high 33% sys).

Was this locally on your machine?  I assume that's without enabling
sanitizers?

In CI the bottleneck clearly is CPU at the moment, due to the relatively now
number of cores.

To reduce IO, one pretty significant thing we can do is to reduce the segment
size used during tests. Creating lots of 16MB segments when most of them are
only very partially used isn't free.



> 0. baseline was ~71s (stuff already hot)
> 1a. down to 64s with dirtywriteback tune (and mostly to avoid NVMe/SSD wear)
> 1b. ~65s with tmpfs, so I've left using dirtywriteback sysctls:
>     sudo mount -t tmpfs -o size=4G,uid=XXX,mode=755  tmpfs build/tmp_install
>     sudo mount -t tmpfs -o size=16G,uid=XXX,mode=755 tmpfs /build/testrun

I don't think we should do that, real FS behaviour is something we do IMO want
to test.


> 2. Splitting the tests (isolation, 027_stream_regress, pg_upgrade) into 4
>    parallel streams of each did not help much (they are longest ones)

> 3. I've spotted the falcon-sensor (EDR agent, using eBPF) very busy, so
>    I've shut it down, got the duratiion down to 43s.

Heh.


> 4. Still for that 43s dominant factor was the mmap/page-fault/PTEs related
>    to the number of backends we spawn. Literally later when I put
>    Claude  to work he said to me this "Backend startup costs roughly 2.5x
>    as much as the actual queries". And later when I've pushed to count using
>    log_connections it said "Got 24,903 total connections in 46 s = 541
>    backend forks/second." and got this top report:
>      8,610   subscription      - 35 % of all connections in the suite
>      4,382   recovery          - 18 %

Hah. I wonder how much of this is just polling for catchup and such. Which we
should totally make smarter (e.g. using WAIT FOR in more places and making
poll_query_until() have adaptive sleep times).


>      1,100   pg_upgrade
>        896   isolation
>        694   pg_dump
>        682   pg_basebackup
> 
>     Fixing above subscription to ~5000 conns did not gain much (well it saved
>     5% of runtime 43s -> 41s). It's literally 10k lines of
>     s/$node_subscriber->safe_psql/sub_bg->query_safe/g across dozens of files
>     in src/test/subscription/t/). Too big for review and I'm not sharing as
>     it could contain errors.

Did you test the effect of those changes on windows (via CI)? I'd expect that
big a reduction to have a substantially bigger effect there.


> 5. Spotted that we do plenty of initdb and cached-initdb (cp), so I had idea
>    about XFS's cp reflinks=always in build/, but I couldn't do that without
>    /dev/loop, so apparently XFS (reflink=1) vs ext4(reflink=0) halves number
>    of writes while even still on /dev/loop device, but that somehow
>    does not directly contribute to duration of the test (well we are
>    bottlenecked on CPU anyway, so this is just smarter? way of avoiding I/O;
>    maybe with cold-caches and on real VMs running with XFS would be faster)
> 
>    +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
>    @@ -687,7 +687,13 @@ sub init
>                   }
>                   else
>                   {
>    -                       @copycmd = qw(cp -RPp);
>    +                       @copycmd = qw(cp --reflink=always -RPp);

Afaict cp uses reflinks automatically by default, if the filesystem supports
it.  On CI it's not supported due to ext4, but locally it seems to work for
me.


> Other interesting ideas: pg_regress with built-in connection pool (IMHO not
> worth it), mitigations=off (to avoid syscalls being taxed, got not
> improvement with this).

I really doubt that the number of connections pg_regress establishes matter in
comparison to the amount of work done per connection in pg_regress style
tests.

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-02T18:43:31Z

Hi,

On 2026-06-02 20:08:17 +0200, Peter Eisentraut wrote:
> On 01.06.26 23:57, Andres Freund wrote:
> > Attached is an large incremental patch onto Bilal's version:
> 
> This looks good to me.

Cool.

> (I suppose the 0005 patch was just for testing.

Correct. Things just take too long to iterate otherwise.

> The 0004 patch looks ok but
> could use some explanation in both the commit message and the code.)

It's also not really for commit at this point. I think we should do it, but it
to help more locally than on CI.

> +  # FIXME: Should we also run on PRs?
> 
> I don't know why we would, since we don't use them.

From what I can tell the workflow of plenty folks during their own development
is to open PRs in their own repo.  I don't really see a downside to also
running on PRs, so I'm inclined to do so. Won't hurt us...

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Jelte Fennema <postgres@jeltef.nl> — 2026-06-02T19:06:39Z

On Tue, Jun 2, 2026, 20:43 Andres Freund <andres@anarazel.de> wrote:

> From what I can tell the workflow of plenty folks during their own
> development
> is to open PRs in their own repo.  I don't really see a downside to also
> running on PRs, so I'm inclined to do so. Won't hurt us...
>

Generally you should do one or the other. Otherwise all jobs will be run
twice when you push to a branch that is linked to a PR. So I'd say, only do
push

>

Re: Heads Up: cirrus-ci is shutting down June 1st

Daniel Gustafsson <daniel@yesql.se> — 2026-06-02T19:12:16Z

> On 2 Jun 2026, at 20:43, Andres Freund <andres@anarazel.de> wrote:

>> +  # FIXME: Should we also run on PRs?
>> 
>> I don't know why we would, since we don't use them.
> 
> From what I can tell the workflow of plenty folks during their own development
> is to open PRs in their own repo.

Seconded, I think it's quite common (I personally do it all the time).

--
Daniel Gustafsson

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-03T00:21:13Z

Hi,

On 2026-06-02 20:36:08 +0300, Nazir Bilal Yavuz wrote:
> You are right, I confirm that this approach solves the problem and I
> think this solution makes sense.

Cool.


> From another thread:
> > I find postgresql-ci.yml a bit long, how about postgres-ci.yml or pg-ci.yml?
>
> I am okay with both but my choice would be postgres-ci.yml.

Due to Peter's input, I went with pg-ci.yml for now. If others want to weigh
in...


I did some more changes:

- Made Meson (64-bit) run address sanitizer, it's about 2 minutes faster

- Rephrased the README

  This is currently done in a separate commit for easier review here, but I
  think I'd merge it before pushing.

- Made the cirrus-ci removal commits more complete

- changed the "name" to "CI for PostgreSQL", the "GitHub Actions" part was
  seemed pointless

- added a small amount of comments at the top of pg-ci.yml

- Some general commit message and comment polishing

- renamed "Windows - VS - Meson & ninja" to "Windows - Visual Studio" the
  former was a bit too long once running and truncated.  Better suggestions
  welcome.

- changed the container URL to ghcr.io/anarazel/pg-vm-images/main, I replaced
  the main branch now.


Think this is pretty close to committable. There's loads more to do, but
getting some CI back seems pretty important, we can iterate subsequently.


Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Peter Eisentraut <peter@eisentraut.org> — 2026-06-03T11:01:49Z

There were a few tests/configurations that the FreeBSD job did under 
Cirrus that we didn't carry over yet.  I looked into this, and the 
attached patch fixes that.  Some of these tests are quite important, I 
think, so we should put those somewhere.

I moved most of them over to macOS, mainly to keep them all together, 
since some things had already been moved there, and the overall run-time 
outcome from this seems reasonable.

The "Test running" step could probably be tweaked cosmetically a bit 
more.  I put in a YAML anchor so we could move it around or duplicate it 
more easily, but maybe that's not necessary or useful (and the use of 
the DYLD_* environment variable doesn't make it portable anyway).  Also, 
it's not clear if the "Stop running server" step is needed and exactly 
how it should be phrased to be most effective in this new environment.

Re: Heads Up: cirrus-ci is shutting down June 1st

Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-03T11:30:01Z

On Wed, Jun 3, 2026 at 2:21 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
[..]

7a-0002 README LGTM

> - changed the container URL to ghcr.io/anarazel/pg-vm-images/main, I replaced
>   the main branch now.

also qq, 7a-0001 has:

    +  # Debian Trixie containers used by all Linux jobs. Built by
    +  # 'https://github.com/anarazel/pg-vm-images/'.
    +  CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/main

$ skopeo inspect docker://ghcr.io/anarazel/pg-vm-images/master:latest
FATA[0000] Error parsing image name
"docker://ghcr.io/anarazel/pg-vm-images/master:latest": reading
manifest latest in ghcr.io/anarazel/pg-vm-images/master: manifest
unknown

and I'm getting 404 with redir to
https://github.com/-/anarazel/packages/container/package/pg-vm-images%2Fmain
(maybe it should be made public to pull those containers?)

-J.

Re: Heads Up: cirrus-ci is shutting down June 1st

Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-03T12:46:56Z

Hi,

On Tue, Jun 2, 2026 at 8:38 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2026-06-01 12:01:58 +0200, Jakub Wartak wrote:
> > So I've spent half of day on trying to see what makes the tests so slow at
> > least in my case. I can also confirm %CPU combined (with high 33% sys).
>
> Was this locally on your machine?  I assume that's without enabling
> sanitizers?

Yup.

> In CI the bottleneck clearly is CPU at the moment, due to the relatively now
> number of cores.
>
> To reduce IO, one pretty significant thing we can do is to reduce the segment
> size used during tests. Creating lots of 16MB segments when most of them are
> only very partially used isn't free.

Right, saw that, nice.

> > 0. baseline was ~71s (stuff already hot)
> > 1a. down to 64s with dirtywriteback tune (and mostly to avoid NVMe/SSD wear)
> > 1b. ~65s with tmpfs, so I've left using dirtywriteback sysctls:
> >     sudo mount -t tmpfs -o size=4G,uid=XXX,mode=755  tmpfs build/tmp_install
> >     sudo mount -t tmpfs -o size=16G,uid=XXX,mode=755 tmpfs /build/testrun
>
> I don't think we should do that, real FS behaviour is something we do IMO want
> to test.

Ack.

> >      1,100   pg_upgrade
> >        896   isolation
> >        694   pg_dump
> >        682   pg_basebackup
> >
> >     Fixing above subscription to ~5000 conns did not gain much (well it saved
> >     5% of runtime 43s -> 41s). It's literally 10k lines of
> >     s/$node_subscriber->safe_psql/sub_bg->query_safe/g across dozens of files
> >     in src/test/subscription/t/). Too big for review and I'm not sharing as
> >     it could contain errors.
>
> Did you test the effect of those changes on windows (via CI)? I'd expect that
> big a reduction to have a substantially bigger effect there.

No I did not and I've wiped the changes already, It was just probe for
any simple
quick wins...

> > 5. Spotted that we do plenty of initdb and cached-initdb (cp), so I had idea
> >    about XFS's cp reflinks=always in build/, but I couldn't do that without
> >    /dev/loop, so apparently XFS (reflink=1) vs ext4(reflink=0) halves number
> >    of writes while even still on /dev/loop device, but that somehow
> >    does not directly contribute to duration of the test (well we are
> >    bottlenecked on CPU anyway, so this is just smarter? way of avoiding I/O;
> >    maybe with cold-caches and on real VMs running with XFS would be faster)
> >
> >    +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
> >    @@ -687,7 +687,13 @@ sub init
> >                   }
> >                   else
> >                   {
> >    -                       @copycmd = qw(cp -RPp);
> >    +                       @copycmd = qw(cp --reflink=always -RPp);
>
> Afaict cp uses reflinks automatically by default, if the filesystem supports
> it.  On CI it's not supported due to ext4, but locally it seems to work for
> me.

Yeah it does, I was just wanted to be double-sure, but then realized with CI
we are on overlay fs on top of host's ext4 :( It's a pitty because that cp could
be instant (even CREATE DATABASE with file_extend_method=clone) as even with
--wal-segsize=1 empty cluster takes ~32MB (3x8MB), but even rough estimates
of even cached initdb calls give huge numbers:

$ grep -r -A 5 'PostgreSQL::Test::Cluster->new' src contrib | grep -Po
'\->init[a-z_]*' | sort | uniq -c
    341 ->init
     98 ->init_from_backup

so that's like 400 * 32MB = 12800 MB? But I get the point of using real fs,
it's just that we should have some option of using throwaway filesystems
(maybe we even do, but on own/dedicated runners).

-J.

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-03T12:53:10Z

Hi,

On 2026-06-03 13:30:01 +0200, Jakub Wartak wrote:
> On Wed, Jun 3, 2026 at 2:21 AM Andres Freund <andres@anarazel.de> wrote:
> > - changed the container URL to ghcr.io/anarazel/pg-vm-images/main, I replaced
> >   the main branch now.
> 
> also qq, 7a-0001 has:
> 
>     +  # Debian Trixie containers used by all Linux jobs. Built by
>     +  # 'https://github.com/anarazel/pg-vm-images/'.
>     +  CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/main
> 
> $ skopeo inspect docker://ghcr.io/anarazel/pg-vm-images/master:latest
> FATA[0000] Error parsing image name
> "docker://ghcr.io/anarazel/pg-vm-images/master:latest": reading
> manifest latest in ghcr.io/anarazel/pg-vm-images/master: manifest
> unknown
> 
> and I'm getting 404 with redir to
> https://github.com/-/anarazel/packages/container/package/pg-vm-images%2Fmain
> (maybe it should be made public to pull those containers?)

That's just the base url, the containers are below it. So it's e.g.
ghcr.io/anarazel/pg-vm-images/main/linux_debian_trixie_ci:latest

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-03T15:11:37Z

Hi,

I found a few more issues with the patches.

- Most of them were around windows, particularly around the package
  installs. I saw a bunch of failures with chocolatey installs failing. Since
  all we used it for is bison, flex and diff, it seems faster and more
  reliable to just install those via msys/pacman.   Takes like 1/10th of the
  time, to boot.

  We don't need to install diff and ninja, those are already installed.

- We were installing openssl 1.1 ourselves, but there's already an existing
  openssl 3.6 on the github runners. So it seems to make sense to just use
  that, instead of spending close to a minute installing it?

- There was also a bunch of error handling missing, because windows shell
  don't have set -e like behavior.

- While tackling these I realized I had dropped meson-logs/ archiving on
  failure.

- Also added a commit to add back a runningcheck test. I see that Peter also
  just posted that, I'll compare with that after this.

  One difference I did see is that I added it to linux meson 32bit, because
  that's the quickest task rn...

- some other boring stuff, see patches for details

Example run: https://github.com/anarazel/postgres/actions/runs/26869491578


I think after merging Peter's and my approaches to runningcheck, we can commit
this. There will be a lot more to do, but this is much better than nothing.



If somebody is looking for a small project to help out with: Right now we run
cpan on every run, without caching. There occasionally are visible network
issues and even without that it takes 30s. So that'd be a good target for
caching. We did that in the past, for cirrus/macos, so that shouldn't be too
hard (c.f. 93d97349461).


Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-03T15:20:00Z

Hi,

On 2026-06-02 21:06:39 +0200, Jelte Fennema-Nio wrote:
> On Tue, Jun 2, 2026, 20:43 Andres Freund <andres@anarazel.de> wrote:
> 
> > From what I can tell the workflow of plenty folks during their own
> > development
> > is to open PRs in their own repo.  I don't really see a downside to also
> > running on PRs, so I'm inclined to do so. Won't hurt us...
> >
> 
> Generally you should do one or the other. Otherwise all jobs will be run
> twice when you push to a branch that is linked to a PR. So I'd say, only do
> push

I think that maybe could be addressed with a bit of conditional logic, by
looking at github.event.pull_request.head.repo or such, and skipping jobs if
it's in the same repo?

But I don't want to experiment with that before adding back basic CI, so I'll
leave this for later.

I also would like to allow to start CI via workflow_call, with inputs for the
container tag to be used, so we could run PG CI after building the containers
used by CI, before tagging them as latest. But that's also seems stuff for
later.

Greetings,

Andres

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-03T15:35:42Z

Hi,

On 2026-06-03 13:01:49 +0200, Peter Eisentraut wrote:
> There were a few tests/configurations that the FreeBSD job did under Cirrus
> that we didn't carry over yet.  I looked into this, and the attached patch
> fixes that.  Some of these tests are quite important, I think, so we should
> put those somewhere.

Agreed.

> I moved most of them over to macOS, mainly to keep them all together, since
> some things had already been moved there, and the overall run-time outcome
> from this seems reasonable.

Given the timings I see, I'd much rather move them to either linux-autoconf or
linux-meson-32. They're nearly twice as fast as macos right now.

Any reason against?

> The "Test running" step could probably be tweaked cosmetically a bit more.

Independent of this, it kinda seems like we should add a meson run target to
run the tests this way... Then this could be triggered more easily.

> I put in a YAML anchor so we could move it around or duplicate it more
> easily, but maybe that's not necessary or useful (and the use of the DYLD_*
> environment variable doesn't make it portable anyway).

That could be solved by setting LD_LIBRARY_PATH via the step's env:...

> Also, it's not clear if the "Stop running server" step is needed and exactly
> how it should be phrased to be most effective in this new environment.

> +      # FIXME This was needed on Cirrus, not clear if still needed on GHA.
> +      - name: Stop running server
> +        if: failure()
> +        run: |
> +          build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop || true
> +

I don't think we need it in the case of failure, it should get terminated on
its own. I've seen that work a bunch of times with cancellations terminating
meson test without terminating the tests first. See e.g.
  https://github.com/anarazel/postgres/actions/runs/26869265453/job/79240470455
where you can see
  Cleaning up orphan processes
  Terminate orphan process: pid (54713) (psql)
  Terminate orphan process: pid (54704) (psql)
  Terminate orphan process: pid (54710) (psql)
  ...

Also, failure() as a condition, without !cancelled(), sometimes causes issues
due to not making the task auto-cancelable, which is annoying...

> From a067f35829f5db7b7b03d2ac1eed934820597a5c Mon Sep 17 00:00:00 2001
> From: Peter Eisentraut <peter@eisentraut.org>
> Date: Wed, 3 Jun 2026 12:47:35 +0200
> Subject: [PATCH v7.2.pe] Move more coverage from previously FreeBSD elsewhere
> 
> The following testing aspects were previously (Cirrus CI) covered by
> the FreeBSD job.  Since we currently (GitHub Actions) don't have
> FreeBSD support, we move these elsewhere for now:
> 
> - RELCACHE_FORCE_RELEASE (moved to macOS)
> - ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS (moved to macOS)
> - PG_TEST_INITDB_EXTRA_OPTS test reading/writing/copying of node trees
>   as well as debug_parallel_query=regress (moved to macOS)
> - PG_TEST_PG_UPGRADE_MODE --link (moved to Linux 64-bit (macOS already
>   tests --clone))

I'd move all of these to linux-autoconf, if you're ok with that?

> - meson test --setup running (moved to macOS)

And this to linux-meson-32?

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-06-03T18:12:43Z

On Tue, Jun 2, 2026 at 11:43 AM Andres Freund <andres@anarazel.de> wrote:
> On 2026-06-02 20:08:17 +0200, Peter Eisentraut wrote:
> > +  # FIXME: Should we also run on PRs?
> >
> > I don't know why we would, since we don't use them.
>
> From what I can tell the workflow of plenty folks during their own development
> is to open PRs in their own repo.  I don't really see a downside to also
> running on PRs, so I'm inclined to do so. Won't hurt us...

I guess I'll pipe up again to mention that we have a lot of downstream
forks. Are we certain that GitHub isn't going to opt them all into
test-every-stable-commit-and-PR on their next sync?

(There was no reply to my previous email on this, so I can't tell if
I'm just way off base. A GitHub discussion on this [1] shows
considerable confusion on how the opt-in occurs; whether it occurs for
all forks, or just new forks after we introduce the workflow; and
whether or not that protection for new forks is itself buggy.)

Thanks,
--Jacob

[1] https://github.com/orgs/community/discussions/26704

Re: Heads Up: cirrus-ci is shutting down June 1st

Peter Eisentraut <peter@eisentraut.org> — 2026-06-03T18:34:41Z

On 03.06.26 17:35, Andres Freund wrote:
>>  From a067f35829f5db7b7b03d2ac1eed934820597a5c Mon Sep 17 00:00:00 2001
>> From: Peter Eisentraut <peter@eisentraut.org>
>> Date: Wed, 3 Jun 2026 12:47:35 +0200
>> Subject: [PATCH v7.2.pe] Move more coverage from previously FreeBSD elsewhere
>>
>> The following testing aspects were previously (Cirrus CI) covered by
>> the FreeBSD job.  Since we currently (GitHub Actions) don't have
>> FreeBSD support, we move these elsewhere for now:
>>
>> - RELCACHE_FORCE_RELEASE (moved to macOS)
>> - ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS (moved to macOS)
>> - PG_TEST_INITDB_EXTRA_OPTS test reading/writing/copying of node trees
>>    as well as debug_parallel_query=regress (moved to macOS)
>> - PG_TEST_PG_UPGRADE_MODE --link (moved to Linux 64-bit (macOS already
>>    tests --clone))
> 
> I'd move all of these to linux-autoconf, if you're ok with that?
> 
> 
>> - meson test --setup running (moved to macOS)
> 
> And this to linux-meson-32?

That seems fine as well.

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-06-03T18:53:26Z

On Wed, Jun 3, 2026 at 8:11 AM Andres Freund <andres@anarazel.de> wrote:
> If somebody is looking for a small project to help out with: Right now we run
> cpan on every run, without caching. There occasionally are visible network
> issues and even without that it takes 30s. So that'd be a good target for
> caching. We did that in the past, for cirrus/macos, so that shouldn't be too
> hard (c.f. 93d97349461).

I will take a look today.

--Jacob

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-03T19:49:31Z

Hi,

On 2026-06-03 20:34:41 +0200, Peter Eisentraut wrote:
> On 03.06.26 17:35, Andres Freund wrote:
> > >  From a067f35829f5db7b7b03d2ac1eed934820597a5c Mon Sep 17 00:00:00 2001
> > > From: Peter Eisentraut <peter@eisentraut.org>
> > > Date: Wed, 3 Jun 2026 12:47:35 +0200
> > > Subject: [PATCH v7.2.pe] Move more coverage from previously FreeBSD elsewhere
> > >
> > > The following testing aspects were previously (Cirrus CI) covered by
> > > the FreeBSD job.  Since we currently (GitHub Actions) don't have
> > > FreeBSD support, we move these elsewhere for now:
> > >
> > > - RELCACHE_FORCE_RELEASE (moved to macOS)
> > > - ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS (moved to macOS)
> > > - PG_TEST_INITDB_EXTRA_OPTS test reading/writing/copying of node trees
> > >    as well as debug_parallel_query=regress (moved to macOS)
> > > - PG_TEST_PG_UPGRADE_MODE --link (moved to Linux 64-bit (macOS already
> > >    tests --clone))
> >
> > I'd move all of these to linux-autoconf, if you're ok with that?
> >
> >
> > > - meson test --setup running (moved to macOS)
> >
> > And this to linux-meson-32?
>
> That seems fine as well.

Done in the attached (0015)

Other changes:

- added a commit to split the tests in windows across two jobs using meson
  test's --slice.  At the expense of the uglier display and use of one more
  runner "slot", it brings down the test times from about 33min to 23min.

  I think that's worth it?


- use ubuntu-24.04 instead of ubuntu-latest, worried that the upgrade will
  break stuff

- formatting and naming polish

- some more missing stored logs and similar stuff


Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-03T19:56:59Z

Hi,

On 2026-06-03 11:12:43 -0700, Jacob Champion wrote:
> On Tue, Jun 2, 2026 at 11:43 AM Andres Freund <andres@anarazel.de> wrote:
> > On 2026-06-02 20:08:17 +0200, Peter Eisentraut wrote:
> > > +  # FIXME: Should we also run on PRs?
> > >
> > > I don't know why we would, since we don't use them.
> >
> > From what I can tell the workflow of plenty folks during their own development
> > is to open PRs in their own repo.  I don't really see a downside to also
> > running on PRs, so I'm inclined to do so. Won't hurt us...
>
> I guess I'll pipe up again to mention that we have a lot of downstream
> forks.

I'm not entirely unconcerned, but I think requiring explicit per-repo
configuration in a postgres specific way will cause more harm long term, than
some folks having to figure out how to disable the workflow.

FWIW, I did add a section about disabling the workflow to src/tools/ci/README.

> Are we certain that GitHub isn't going to opt them all into
> test-every-stable-commit-and-PR on their next sync?

It'd not test every stable commit, just the ones separately pushed, no?

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-06-03T23:03:20Z

On Wed, Jun 3, 2026 at 12:57 PM Andres Freund <andres@anarazel.de> wrote:
> On 2026-06-03 11:12:43 -0700, Jacob Champion wrote:
> > Are we certain that GitHub isn't going to opt them all into
> > test-every-stable-commit-and-PR on their next sync?
>
> It'd not test every stable commit, just the ones separately pushed, no?

Ah. Slightly embarrassing: I misunderstood the Sync Fork functionality
in GitHub, which I'd never actually used. It's a one-time
synchronization, not a permanent "keep this up to date" toggle, so the
situation's not as alarmingly carbon-intensive as I made it sound.

So yes, just every push. I don't know if the Sync Fork button acts as
a push trigger as well.

> > I guess I'll pipe up again to mention that we have a lot of downstream
> > forks.
>
> I'm not entirely unconcerned, but I think requiring explicit per-repo
> configuration in a postgres specific way will cause more harm long term

What kind of harm are we talking about -- just that they have to
follow the steps that our hypothetical skip logic prints out, or else
ask on the list?

> than
> some folks having to figure out how to disable the workflow.

A vanishingly small percentage of our 5,700 forks have to sync up with
us in order to permanently outnumber the people who will ever test PRs
on purpose, I think.

Concretely: I propose that we bail out of the setup step if the
repository isn't postgres/postgres, just for the initial committed
version, and then we can test what this actually does in practice to
our downstream forks. If I'm being overly paranoid, we can immediately
remove it; else we can add an opt-in. But adding it after the fact
won't protect anyone who synced up in the interim.

--Jacob

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-03T23:36:29Z

Hi,

On 2026-06-03 16:03:20 -0700, Jacob Champion wrote:
> On Wed, Jun 3, 2026 at 12:57 PM Andres Freund <andres@anarazel.de> wrote:
> > On 2026-06-03 11:12:43 -0700, Jacob Champion wrote:
> > > Are we certain that GitHub isn't going to opt them all into
> > > test-every-stable-commit-and-PR on their next sync?
> >
> > It'd not test every stable commit, just the ones separately pushed, no?
> 
> Ah. Slightly embarrassing: I misunderstood the Sync Fork functionality
> in GitHub, which I'd never actually used. It's a one-time
> synchronization, not a permanent "keep this up to date" toggle, so the
> situation's not as alarmingly carbon-intensive as I made it sound.
> 
> So yes, just every push. I don't know if the Sync Fork button acts as
> a push trigger as well.

Jacob and I just tested this with a test account that I had around.  A new
fork starts out with disabled workflows. But forking before this and then
resyncing / pulling remaining changes, does lead to the workflow being
disabled.


> > > I guess I'll pipe up again to mention that we have a lot of downstream
> > > forks.
> >
> > I'm not entirely unconcerned, but I think requiring explicit per-repo
> > configuration in a postgres specific way will cause more harm long term
> 
> What kind of harm are we talking about -- just that they have to
> follow the steps that our hypothetical skip logic prints out, or else
> ask on the list?

Yes.  I spent a decent chunk of time helping folks set up CI for
cirrus-ci. And yet there continued to be folks that were surprised you could
run CI for yourself.



> Concretely: I propose that we bail out of the setup step if the
> repository isn't postgres/postgres, just for the initial committed
> version, and then we can test what this actually does in practice to
> our downstream forks. If I'm being overly paranoid, we can immediately
> remove it; else we can add an opt-in. But adding it after the fact
> won't protect anyone who synced up in the interim.

We clearly would need to have an opt-out from that from the get-go, otherwise
I couldn't even test that things are still working before merging, and
postgresql-cfbot won't work...  Thomas has it otherwise mostly ready to go
(this thread actually is being tested automatically already).

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-03T23:40:45Z

Hi,

On 2026-06-03 19:36:29 -0400, Andres Freund wrote:
> On 2026-06-03 16:03:20 -0700, Jacob Champion wrote:
> > On Wed, Jun 3, 2026 at 12:57 PM Andres Freund <andres@anarazel.de> wrote:
> > > On 2026-06-03 11:12:43 -0700, Jacob Champion wrote:
> > > > Are we certain that GitHub isn't going to opt them all into
> > > > test-every-stable-commit-and-PR on their next sync?
> > >
> > > It'd not test every stable commit, just the ones separately pushed, no?
> > 
> > Ah. Slightly embarrassing: I misunderstood the Sync Fork functionality
> > in GitHub, which I'd never actually used. It's a one-time
> > synchronization, not a permanent "keep this up to date" toggle, so the
> > situation's not as alarmingly carbon-intensive as I made it sound.
> > 
> > So yes, just every push. I don't know if the Sync Fork button acts as
> > a push trigger as well.
> 
> Jacob and I just tested this with a test account that I had around.  A new
> fork starts out with disabled workflows. But forking before this and then
> resyncing / pulling remaining changes, does lead to the workflow being
> disabled.

Err, I typo'd, thinko'd the last disabled, that should have been "enabled",
unfortunately.

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-04T02:46:14Z

Hi,

On 2026-06-03 19:36:29 -0400, Andres Freund wrote:
> > Concretely: I propose that we bail out of the setup step if the
> > repository isn't postgres/postgres, just for the initial committed
> > version, and then we can test what this actually does in practice to
> > our downstream forks. If I'm being overly paranoid, we can immediately
> > remove it; else we can add an opt-in. But adding it after the fact
> > won't protect anyone who synced up in the interim.
> 
> We clearly would need to have an opt-out from that from the get-go, otherwise
> I couldn't even test that things are still working before merging, and
> postgresql-cfbot won't work...  Thomas has it otherwise mostly ready to go
> (this thread actually is being tested automatically already).

Attached is a possible implementation of this.  If the PG_CI_ENABLED
repository variable is not set to 1, we run as little CI as possible.

To make that less confusing, emit a summary whenever we skip running CI, with
a message explaining how to enable CI.  See e.g. the bottom of
https://github.com/anarazel/postgres/actions/runs/26926523027

Unfortunately this summary is only visible when logged into github.

Attached is a squashed version of my earlier changes. In addition:

- src/tools/ci/README was updated with the knowledge that the workflow
  may or may not be enabled in a fork

- a few typos etc were fixed

- the change above was added as a third patch

I'm on the fence whether we want the third change or not.

I'd like to push the CI support tomorrow morning.  Perhaps othes will chime
about whether we should require this explicit opt-in or not...

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-06-04T11:12:32Z

Hi,

On Thu, 4 Jun 2026 at 05:46, Andres Freund <andres@anarazel.de> wrote:
> On 2026-06-03 19:36:29 -0400, Andres Freund wrote:
> > > Concretely: I propose that we bail out of the setup step if the
> > > repository isn't postgres/postgres, just for the initial committed
> > > version, and then we can test what this actually does in practice to
> > > our downstream forks. If I'm being overly paranoid, we can immediately
> > > remove it; else we can add an opt-in. But adding it after the fact
> > > won't protect anyone who synced up in the interim.
> >
> > We clearly would need to have an opt-out from that from the get-go, otherwise
> > I couldn't even test that things are still working before merging, and
> > postgresql-cfbot won't work...  Thomas has it otherwise mostly ready to go
> > (this thread actually is being tested automatically already).
>
>
> Attached is a possible implementation of this.  If the PG_CI_ENABLED
> repository variable is not set to 1, we run as little CI as possible.
>
> To make that less confusing, emit a summary whenever we skip running CI, with
> a message explaining how to enable CI.  See e.g. the bottom of
> https://github.com/anarazel/postgres/actions/runs/26926523027
>
> Unfortunately this summary is only visible when logged into github.
>
>
> Attached is a squashed version of my earlier changes. In addition:
>
> - src/tools/ci/README was updated with the knowledge that the workflow
>   may or may not be enabled in a fork
>
> - a few typos etc were fixed
>
> - the change above was added as a third patch
>
>
> I'm on the fence whether we want the third change or not.

0001 LGTM. I really liked the slicing solution on Windows VS.


0002:

Nitpick: There are two mentions of GitHub Actions with lowercase in
the 'ci_macports_packages.sh', perhaps we can update them with 'GitHub
Actions'.


0003:

I think it makes sense to be explicit about this. I don't think this
CI will cause any problems (to users) since Github Actions is free on
public repositories but it is better safe than sorry. Also, from
discussion on PostgreSQL Hacking Discord channel:

Thomas Munro mentioned:

 > Maybe "Report if CI is not enabled in this repository" would be
slightly clearer.  it's not really a warning, it's just stating a
fact.  or maybe "Report if PG_CI_ENABLED is not set to 1 in this
repository" even?

I liked 'Report if CI is not enabled in this repository'.

Jelte Fennema-Nio mentioned:

> I think being opt-in is fine. I'd actually fail the build though, instead of succeeding. People are probably more inclined to click the red cross next to a commit than a green checkmark.

I think this makes sense too. Green color can be seen as it is
expected and nothing wrong.

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-06-04T12:13:14Z

Hi,

On Thu, 4 Jun 2026 at 14:12, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> >
> > To make that less confusing, emit a summary whenever we skip running CI, with
> > a message explaining how to enable CI.  See e.g. the bottom of
> > https://github.com/anarazel/postgres/actions/runs/26926523027

I think it would be nice to mention that we need to create a
'Repository variable' not an 'Environment variable'.

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Peter Eisentraut <peter@eisentraut.org> — 2026-06-04T13:27:42Z

On 04.06.26 04:46, Andres Freund wrote:
> Hi,
> 
> On 2026-06-03 19:36:29 -0400, Andres Freund wrote:
>>> Concretely: I propose that we bail out of the setup step if the
>>> repository isn't postgres/postgres, just for the initial committed
>>> version, and then we can test what this actually does in practice to
>>> our downstream forks. If I'm being overly paranoid, we can immediately
>>> remove it; else we can add an opt-in. But adding it after the fact
>>> won't protect anyone who synced up in the interim.
>>
>> We clearly would need to have an opt-out from that from the get-go, otherwise
>> I couldn't even test that things are still working before merging, and
>> postgresql-cfbot won't work...  Thomas has it otherwise mostly ready to go
>> (this thread actually is being tested automatically already).
> 
> 
> Attached is a possible implementation of this.  If the PG_CI_ENABLED
> repository variable is not set to 1, we run as little CI as possible.
> 
> To make that less confusing, emit a summary whenever we skip running CI, with
> a message explaining how to enable CI.  See e.g. the bottom of
> https://github.com/anarazel/postgres/actions/runs/26926523027

I think this would be a great solution.  The message is easy to find and 
the instructions are easy to follow.

I suggest maybe using "Note" instead of "Caution" because otherwise it 
sounds quite dangerous and maybe we want to reserve that for when we 
have some serious things to report.  Also I suggest writing "PostgreSQL 
CI is disabled ...", not just "CI is disabled ...", since that might 
otherwise be confusing, especially since this is being considered in the 
context of downstream forks that were not expecting this.  (I also 
considered "PostgreSQL community CI ..." to be even more explicit.)

> Attached is a squashed version of my earlier changes. In addition:
> 
> - src/tools/ci/README was updated with the knowledge that the workflow
>    may or may not be enabled in a fork
> 
> - a few typos etc were fixed
> 
> - the change above was added as a third patch

Since you made a point of reordering dependencies, the 
${MINGW_PACKAGE_PREFIX}-zstd is not quite in the right place.

> I'm on the fence whether we want the third change or not.

This seems useful.  Otherwise, if you have a private fork and just want 
to park some patches, this will blow through your private repository 
free credits in about 5 or 6 builds.  (Learned that a few days ago. ;-) )

Re: Heads Up: cirrus-ci is shutting down June 1st

Peter Eisentraut <peter@eisentraut.org> — 2026-06-04T13:28:46Z

On 04.06.26 14:13, Nazir Bilal Yavuz wrote:
> Hi,
> 
> On Thu, 4 Jun 2026 at 14:12, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>>
>>>
>>> To make that less confusing, emit a summary whenever we skip running CI, with
>>> a message explaining how to enable CI.  See e.g. the bottom of
>>> https://github.com/anarazel/postgres/actions/runs/26926523027
> 
> I think it would be nice to mention that we need to create a
> 'Repository variable' not an 'Environment variable'.

Yes, it wasn't clear to me what the difference was or which one I should 
choose.  (Your message helped.)

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-04T14:04:11Z

Hi,

On 2026-06-04 14:12:32 +0300, Nazir Bilal Yavuz wrote:
> 0001 LGTM. I really liked the slicing solution on Windows VS.

Cool.

> 0002:
> 
> Nitpick: There are two mentions of GitHub Actions with lowercase in
> the 'ci_macports_packages.sh', perhaps we can update them with 'GitHub
> Actions'.

done.

> 0003:
> 
> I think it makes sense to be explicit about this. I don't think this
> CI will cause any problems (to users) since Github Actions is free on
> public repositories but it is better safe than sorry.

It turns out that if you make a private fork of the public PG repo, it's
rather easy to churn through the free minutes included in the account...

But I think Jacob's concern is more that there's thousands of forks of
postgres, if they all suddenly start to run CI, that's quite a bit of wasted
effort (including downloading our containers etc).

> Also, from discussion on PostgreSQL Hacking Discord channel:
> 
> Thomas Munro mentioned:
> 
>  > Maybe "Report if CI is not enabled in this repository" would be
> slightly clearer.  it's not really a warning, it's just stating a
> fact.  or maybe "Report if PG_CI_ENABLED is not set to 1 in this
> repository" even?

For a moment I thought we could solve this by using ${{case(...)} in the job's
.name, but it turns out those are only evaluated when the job gets closer
would have a chance to run. So it ends up looking too ugly.

> I liked 'Report if CI is not enabled in this repository'.

After Thomas' suggestion I first went with:
  "Report if repo has not opted into CI"
but that turned out to be truncated in the display. So now I went with
  "Report if not opted into CI"
which is not truncated.

How is that?

> Jelte Fennema-Nio mentioned:
>
> > I think being opt-in is fine. I'd actually fail the build though, instead of succeeding. People are probably more inclined to click the red cross next to a commit than a green checkmark.
> 
> I think this makes sense too. Green color can be seen as it is
> expected and nothing wrong.

I don't like it much, because it means suddenly a lot of red will appear in
people that just have a fork to look at code etc...

Downthread you wrote:

On 2026-06-04 15:13:14 +0300, Nazir Bilal Yavuz wrote:
> On Thu, 4 Jun 2026 at 14:12, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> >
> > >
> > > To make that less confusing, emit a summary whenever we skip running CI, with
> > > a message explaining how to enable CI.  See e.g. the bottom of
> > > https://github.com/anarazel/postgres/actions/runs/26926523027
> 
> I think it would be nice to mention that we need to create a
> 'Repository variable' not an 'Environment variable'.

Updated, both in the message and the README.

Nearby:

On 2026-06-04 15:27:42 +0200, Peter Eisentraut wrote:
> On 04.06.26 04:46, Andres Freund wrote:
> > Attached is a possible implementation of this.  If the PG_CI_ENABLED
> > repository variable is not set to 1, we run as little CI as possible.
> > 
> > To make that less confusing, emit a summary whenever we skip running CI, with
> > a message explaining how to enable CI.  See e.g. the bottom of
> > https://github.com/anarazel/postgres/actions/runs/26926523027
> 
> I think this would be a great solution.  The message is easy to find and the
> instructions are easy to follow.

Cool.

It's not great that it doesn't show up if one isn't logged in, but I think
it's about as good as we can do.

> I suggest maybe using "Note" instead of "Caution" because otherwise it
> sounds quite dangerous and maybe we want to reserve that for when we have
> some serious things to report.

I changed it to IMPORTANT, that seemed more fitting than just a NOTE?

> Also I suggest writing "PostgreSQL CI is disabled ...", not just "CI is
> disabled ...", since that might otherwise be confusing, especially since
> this is being considered in the context of downstream forks that were not
> expecting this.

Done.

> (I also considered "PostgreSQL community CI ..." to be even more explicit.)

That feels a bit too much. I made it ${{github.workflow}}, which is the name
set at the top of the workflow.

I also changed to use tee -a for the printing the opt-in summary, so the
message is visible inside the step, as otherwise one doesn't see anything when
navigating to the "Report ..."  job. It's just markdown printed literally, but
it seems readable enough.

I'm a bit torn about "opted into in this repository" vs "opted into for this
repository". Any language nerds ready for action?

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-06-04T14:28:30Z

Hi,

On Thu, 4 Jun 2026 at 17:04, Andres Freund <andres@anarazel.de> wrote:
>
> On 2026-06-04 14:12:32 +0300, Nazir Bilal Yavuz wrote:
> > 0001 LGTM. I really liked the slicing solution on Windows VS.
>
> > 0003:
> >
> > I think it makes sense to be explicit about this. I don't think this
> > CI will cause any problems (to users) since Github Actions is free on
> > public repositories but it is better safe than sorry.
>
> It turns out that if you make a private fork of the public PG repo, it's
> rather easy to churn through the free minutes included in the account...

Yes, it sounds bad :(


> But I think Jacob's concern is more that there's thousands of forks of
> postgres, if they all suddenly start to run CI, that's quite a bit of wasted
> effort (including downloading our containers etc).

I got it, yes that is a valid concern.


> > Also, from discussion on PostgreSQL Hacking Discord channel:
> >
> > Thomas Munro mentioned:
> >
> >  > Maybe "Report if CI is not enabled in this repository" would be
> > slightly clearer.  it's not really a warning, it's just stating a
> > fact.  or maybe "Report if PG_CI_ENABLED is not set to 1 in this
> > repository" even?
>
> For a moment I thought we could solve this by using ${{case(...)} in the job's
> .name, but it turns out those are only evaluated when the job gets closer
> would have a chance to run. So it ends up looking too ugly.
>
>
> > I liked 'Report if CI is not enabled in this repository'.
>
> After Thomas' suggestion I first went with:
>   "Report if repo has not opted into CI"
> but that turned out to be truncated in the display. So now I went with
>   "Report if not opted into CI"
> which is not truncated.
>
> How is that?

I think this is better. I am okay with 'Report if not opted into CI'.


> > Jelte Fennema-Nio mentioned:
> >
> > > I think being opt-in is fine. I'd actually fail the build though, instead of succeeding. People are probably more inclined to click the red cross next to a commit than a green checkmark.
> >
> > I think this makes sense too. Green color can be seen as it is
> > expected and nothing wrong.
>
> I don't like it much, because it means suddenly a lot of red will appear in
> people that just have a fork to look at code etc...

I see, yes you are right. People might start to get emails about the
CI failing if we would implement it like that and that would be a
frustrating experience.


> On 2026-06-04 15:27:42 +0200, Peter Eisentraut wrote:
> > On 04.06.26 04:46, Andres Freund wrote:
> > > Attached is a possible implementation of this.  If the PG_CI_ENABLED
> > > repository variable is not set to 1, we run as little CI as possible.
>
> > I suggest maybe using "Note" instead of "Caution" because otherwise it
> > sounds quite dangerous and maybe we want to reserve that for when we have
> > some serious things to report.
>
> I changed it to IMPORTANT, that seemed more fitting than just a NOTE?

I think 'IMPORTANT' is better compared to 'NOTE'.


> I also changed to use tee -a for the printing the opt-in summary, so the
> message is visible inside the step, as otherwise one doesn't see anything when
> navigating to the "Report ..."  job. It's just markdown printed literally, but
> it seems readable enough.

This makes sense. That was the first thing I did when I was testing.


-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-04T15:25:50Z

Hi,

One more revision:

Attached are the prior patches plus one incremental one (to be squashed),
making the macports caching a bit smarter.

The performance improvements after touching
src/tools/ci/ci_macports_packages.sh, updating the package list, or starting
with a clean cache and then failing during build/test seems clearly worth the
complexity.

Greetings,

Andres

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-06-04T15:36:04Z

On Thu, Jun 4, 2026 at 8:25 AM Andres Freund <andres@anarazel.de> wrote:
> Attached are the prior patches plus one incremental one (to be squashed),
> making the macports caching a bit smarter.

> 1) We should save the macports cache before building & running tests, we don't
>   want to again start from scratch if the task failed or was cancelled

+1, thank you! You beat me to the suggestion.

On the CPAN front, my first attempt tried to do too much, so I'm
testing a solution that just caches the CPAN sources directory. Should
have something shortly.

Thanks,
--Jacob

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-06-04T15:50:20Z

Hi,

On Thu, 4 Jun 2026 at 18:25, Andres Freund <andres@anarazel.de> wrote:
>
> One more revision:
>
> Attached are the prior patches plus one incremental one (to be squashed),
> making the macports caching a bit smarter.
>
> The performance improvements after touching
> src/tools/ci/ci_macports_packages.sh, updating the package list, or starting
> with a clean cache and then failing during build/test seems clearly worth the
> complexity.

From v12a-0003 commit message:

1) We should save the macports cache before building & running tests, we don't
   want to again start from scratch if the task failed or was cancelled

2) We should use a partial cache match, as it's much faster to start from
   that, than from scratch.

Also, 'not saving the cache if it is an exact hit' is a nice improvement too.

I think these are all great improvements. I looked at the 0003 and LGTM.

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-04T16:00:15Z

Hi,

On 2026-06-04 08:36:04 -0700, Jacob Champion wrote:
> On Thu, Jun 4, 2026 at 8:25 AM Andres Freund <andres@anarazel.de> wrote:
> > Attached are the prior patches plus one incremental one (to be squashed),
> > making the macports caching a bit smarter.
> 
> > 1) We should save the macports cache before building & running tests, we don't
> >   want to again start from scratch if the task failed or was cancelled
> 
> +1, thank you! You beat me to the suggestion.

Cool.


> On the CPAN front, my first attempt tried to do too much, so I'm
> testing a solution that just caches the CPAN sources directory. Should
> have something shortly.

I'm kinda inclined to push what we have and then add the cpan caching on top
in subsequent commits?

I'm pretty sure we're going to find a bunch of problems once cfbot starts
hammering, so it's not like we're otherwise not going to need to touch this
further.

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-06-04T16:02:13Z

On Thu, Jun 4, 2026 at 9:00 AM Andres Freund <andres@anarazel.de> wrote:
> > On the CPAN front, my first attempt tried to do too much, so I'm
> > testing a solution that just caches the CPAN sources directory. Should
> > have something shortly.
>
> I'm kinda inclined to push what we have and then add the cpan caching on top
> in subsequent commits?

Right -- I don't think v1 should wait on CPAN optimization; I just
wanted to let you know the status.

--Jacob

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-04T19:03:39Z

Hi,

On 2026-06-04 09:02:13 -0700, Jacob Champion wrote:
> On Thu, Jun 4, 2026 at 9:00 AM Andres Freund <andres@anarazel.de> wrote:
> > > On the CPAN front, my first attempt tried to do too much, so I'm
> > > testing a solution that just caches the CPAN sources directory. Should
> > > have something shortly.
> >
> > I'm kinda inclined to push what we have and then add the cpan caching on top
> > in subsequent commits?
> 
> Right -- I don't think v1 should wait on CPAN optimization; I just
> wanted to let you know the status.

Pushed it now. Only a tiny change from the last version, Bilal suggested ove
IM to remove the multi-threading argument from robocopy, for performance.

I did futz around with ccache improvements for a bit. I think we're going to
need them, but they're complicated enough to do them separately.

Thanks to Bilal, Jelte and all the review!

Greetings,

Andres

Re: Heads Up: cirrus-ci is shutting down June 1st

Michael Paquier <michael@paquier.xyz> — 2026-06-05T05:12:31Z

On Thu, Jun 04, 2026 at 03:03:39PM -0400, Andres Freund wrote:
> Thanks to Bilal, Jelte and all the review!

Woohoo.  Thanks everybody for this thread.  I have just enabled it in
my dev github repo, and that looks to work very nicely!
--
Michael

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-06-05T15:54:44Z

Hi,

On Thu, 4 Jun 2026 at 22:03, Andres Freund <andres@anarazel.de> wrote:
>
> On 2026-06-04 09:02:13 -0700, Jacob Champion wrote:
> > On Thu, Jun 4, 2026 at 9:00 AM Andres Freund <andres@anarazel.de> wrote:
> > > > On the CPAN front, my first attempt tried to do too much, so I'm
> > > > testing a solution that just caches the CPAN sources directory. Should
> > > > have something shortly.
> > >
> > > I'm kinda inclined to push what we have and then add the cpan caching on top
> > > in subsequent commits?
> >
> > Right -- I don't think v1 should wait on CPAN optimization; I just
> > wanted to let you know the status.
>
> Pushed it now. Only a tiny change from the last version, Bilal suggested ove
> IM to remove the multi-threading argument from robocopy, for performance.

Thank you so much!

I realized one small problem today. When both Windows VS jobs fail, we
end up having two artifacts with the exactly same name. Here is a
small fix for it by adding '-slice-${slice_num}' to the end of the
artifact name.

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-09T10:32:31Z

Hi Nazir/Andres,

On Tue, Jun 2, 2026 at 12:13 PM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
>
> Hi Andres/Nazir,
>
[..]
> Continuing on previous story...:
> Windows was still @ 31mins, and whatever I've tried it is was not helping it
> (but I cannot measure inside GHA Runner what was happening, so those were blind
> shots with fstweaks, etc). One important thing, altough I failed altering
> CacheIsPowerProtected (avoid flushing the write cache) as it seems impossible
> for me to do so on D:\ (as paging file is there and and altering it also
> requires reboot), at least we know stuff is way slower than it could be on
> those runners:
>
> "Get-PhysicalDisk | Get-StorageAdvancedProperty" reported:
>
> FriendlyName      SerialNumber IsPowerProtected IsDeviceCacheEnabled
> ------------      ------------ ---------------- --------------------
> Msft Virtual Disk                         False                False
> Msft Virtual Disk                         False                False
>
> Perhaps there's way to use some custom image/templ with different settings,
> especially for D:\, after all it's just volatile stuff. Thoughts? (not that I
> care that much for Win, but waiting half hour for it finish every time is
> not going to be nice...)
>
[..]

OK, so to close the loop: does no no-write-flushing (and ReFS) can help us here?
I've made it work, but the possible configuration is just slower (just
"Test run"
step) by +2mins (26vs28 mins) :(

Longer:
* This is windows 2022 server, so ReFS (MS next-gen fs) is available.
Technically
  robocopy should do CoW (for our initdb clones out there).
* D:\ cannot cannot be reformatted from NTFS as ReFS mainly due to
active pagefile
  and github agent places files there too.
* But (!) one can make loop-image on D:\ with ReFS (sic!)
* And disable write-cache-flushing with some hacks (usually used with
RAID cards with
  BBU)

And I've bumped TEST_JOBS 4->8 (even with 4 VCPUs), because my local
runs showed in
taskmgr that after quite some time we have ended up using just ~40%
CPU (also with
4 VCPUs) while not doing I/O (this is somehow contrary to what Andres
was stating
earlier). I cannot find way to add observability of CPU usage on GHA runner, so
just gonna leave it as that (but before anybody wishes to add more CPU it would
actually help if such workload on GHA is really on CPU or I/O there).

So it appears that without going into the dragon's den (I mean deeply
analyzing our
tests, especially subscription and recovery), we won't gain much in such setup.

Patch attached if anybody wants to experiment more.

-J.

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-09T12:14:20Z

Hi,

On 2026-06-09 12:32:31 +0200, Jakub Wartak wrote:
> And I've bumped TEST_JOBS 4->8 (even with 4 VCPUs), because my local runs
> showed in taskmgr that after quite some time we have ended up using just
> ~40% CPU (also with 4 VCPUs) while not doing I/O (this is somehow contrary
> to what Andres was stating earlier).

FWIW, I only measured this for linux, not for windows. On linux it was easy to
do

+          vmstat -y -n -w 1 > vmstat.log &
+
+          meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}}
+
+          killall iostat vmstat || true
+
+      - name: Upload stats
+        uses: actions/upload-artifact@v7
+        with:
+          path: |
+            iostat.log
+            vmstat.log

Which showed that there is very little idle CPU other than during first few
seconds and at the end.

I don't know how to do that on windows...  I'm sure one can do it, with ETW or
such, but...

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-10T11:13:49Z

On Tue, Jun 9, 2026 at 2:14 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2026-06-09 12:32:31 +0200, Jakub Wartak wrote:
> > And I've bumped TEST_JOBS 4->8 (even with 4 VCPUs), because my local
runs
> > showed in taskmgr that after quite some time we have ended up using just
> > ~40% CPU (also with 4 VCPUs) while not doing I/O (this is somehow
contrary
> > to what Andres was stating earlier).
>
> FWIW, I only measured this for linux, not for windows. On linux it was
easy to
> do
>
> +          vmstat -y -n -w 1 > vmstat.log &
> +
> +          meson test ${{env.MTEST_ARGS}} --num-processes
${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}}
> +
> +          killall iostat vmstat || true
> +
> +      - name: Upload stats
> +        uses: actions/upload-artifact@v7
> +        with:
> +          path: |
> +            iostat.log
> +            vmstat.log
>
> Which showed that there is very little idle CPU other than during first
few
> seconds and at the end.
>
> I don't know how to do that on windows...  I'm sure one can do it, with
ETW or
> such, but...

Oh, I thought you guys there were have some secret keys to access GH(MS)
stuff :)

OK, so I've gathered similiar data (using "typeperf", learned that the
thing
exists just today). The 'Test world' steps took 14min and 11min (due to
that matrix split, cool trick btw). Attached are: patch how it was gathered,
raw CSV data, and most importantly graph.

We were both right and wrong. It is either CPU bottleneck, but also
if the I/O is involved the CPU drops to <20% in case of runner #1 (same
happens with runner#2 but for short time of 2 mins). Pretty much had
similiar local Windows behavior.

IMHO *if* we want to push that faster it would make some sense to eliminate
that I/O (but after observing that matrix split trich I'm not so sure if it
is worth investing more into it). We seem to drop CPU use every time the
avg disk queue len >= 2.

Alvaro had an idea here in [1] about instance reusing. Or maybe offload that
and ask GH folks to provide images with XFS and ReFS on D:\ by default
instead
?

-J.

[1] -
https://www.postgresql.org/message-id/ah2QDZyOKgW9yU9D%40alvherre.pgsql

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-10T14:12:26Z

Hi,

On 2026-06-10 13:13:49 +0200, Jakub Wartak wrote:
> On Tue, Jun 9, 2026 at 2:14 PM Andres Freund <andres@anarazel.de> wrote:
> > On 2026-06-09 12:32:31 +0200, Jakub Wartak wrote:
> > > And I've bumped TEST_JOBS 4->8 (even with 4 VCPUs), because my local
> runs
> > > showed in taskmgr that after quite some time we have ended up using just
> > > ~40% CPU (also with 4 VCPUs) while not doing I/O (this is somehow
> contrary
> > > to what Andres was stating earlier).
> >
> > FWIW, I only measured this for linux, not for windows. On linux it was
> easy to
> > do
> >
> > +          vmstat -y -n -w 1 > vmstat.log &
> > +
> > +          meson test ${{env.MTEST_ARGS}} --num-processes
> ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}}
> > +
> > +          killall iostat vmstat || true
> > +
> > +      - name: Upload stats
> > +        uses: actions/upload-artifact@v7
> > +        with:
> > +          path: |
> > +            iostat.log
> > +            vmstat.log
> >
> > Which showed that there is very little idle CPU other than during first
> few
> > seconds and at the end.
> >
> > I don't know how to do that on windows...  I'm sure one can do it, with
> ETW or
> > such, but...
>
> Oh, I thought you guys there were have some secret keys to access GH(MS)
> stuff :)

I don't, and I doubt I could have. Nor would I even know where to ask :)

> OK, so I've gathered similiar data (using "typeperf", learned that the thing
> exists just today).

Nice.

> Attached are: patch how it was gathered, raw CSV data, and most importantly
> graph.

Looking at the raw data, I think something must not be quite right. Note how
low the absolute read/write IO numbers are. Is it possible that that's for the
C:/ disk, but that we're doing IO on D:/?

How exctly did you translate the csv data to %cpu utilization?

> We were both right and wrong. It is either CPU bottleneck, but also
> if the I/O is involved the CPU drops to <20% in case of runner #1 (same
> happens with runner#2 but for short time of 2 mins). Pretty much had
> similiar local Windows behavior.

Kinda looks like what we might want is to increase the times / amounts
equivalent to
/proc/sys/vm/{dirty_expire_centisecs,dirty_writeback_centisecs,dirty_background_ratio,dirty_ratio}

But due to the issue mentioned above, I'm not sure we can conclude that much
yet.

> IMHO *if* we want to push that faster it would make some sense to eliminate
> that I/O (but after observing that matrix split trich I'm not so sure if it
> is worth investing more into it). We seem to drop CPU use every time the
> avg disk queue len >= 2.

I'm not that concerned about the VS runtime right now, due to the split, but
mingw very frequently is the slowest task (with an empty / inapplicable cache
it's compilerwarnings, but I have some pending improvements for that, by
converting it to meson the worst case time halves). We can't just split all
tasks, that uses too many of the available "job slots".

> Or maybe offload that and ask GH folks to provide images with XFS and ReFS
> on D:\ by default instead ?

I suspect that will be a very heavy lift.  That'd be a large change and there
are lot of users of this stuff.

It's probably worth seeing what the times with a newer windows image are,
before we do much more.

> Alvaro had an idea here in [1] about instance reusing.

We have the ability to run instances against a running cluster already, but
only use that in one place. I was wondering about a meson test "setup" that
will only run tests that can *not* be run against a running instance.

With a bit of additional scripting (we need the ability to set LD_LIBRARY_PATH
in a cross platform, we have that in a bunch of places, just need to expose
it), that'd allow us to convert all the meson based tests to use the running
tests, and all the tests that don't support that, without duplication between
the runs.

It's not really a fair comparison (due to what's running concurrently), but
here's the time for a few tests in running and a dedicated cluster:

70/398 postgresql:bloom / bloom/regress                                           OK                1.97s   1 subtests passed
6/88 postgresql:bloom-running / bloom-running/regress                             OK                0.54s   1 subtests passed

68/398 postgresql:auto_explain / auto_explain/regress                             OK                1.96s   2 subtests passed
5/88 postgresql:auto_explain-running / auto_explain-running/regress               OK                0.33s   2 subtests passed

77/398 postgresql:cube / cube/regress                                             OK                2.27s   2 subtests passed
11/88 postgresql:cube-running / cube-running/regress                              OK                0.84s   2 subtests passed

Clearly we could gain some if we we didn't run the tests that supported
running against an existing cluster against separate clusters each.

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-10T14:55:21Z

Hi,

On 2026-06-04 15:03:39 -0400, Andres Freund wrote:
> Pushed it now. Only a tiny change from the last version, Bilal suggested ove
> IM to remove the multi-threading argument from robocopy, for performance.
>
> I did futz around with ccache improvements for a bit. I think we're going to
> need them, but they're complicated enough to do them separately.

The ccache improvements have been committed since, in:
2026-06-08 f52c44ce48a ci: Improve ccache handling

Before we can backpatch the CI support I think we need to resolve a few more
things:

- re-enabling crash reporting for windows, Bilal sent a patch [1]

I think that's pretty much a must have, otherwise debuggin windows issues is
really hard.

- Cold or inapplicable (e.g. due to a core header change) compiler warnings
task is very slow (35min). I have a patch that I need to send out to
convert everything but the headercheck in compilerwarnings to meson, that
reduces the worst case build times considerably (primarily due to the cross
build being able to use precompiled headers)

I'll try to send that out later today.

There's other issues, but I'm not sure we need to resolve them before
backpatching:

- Coverage for the BSDs - this is complicated enough that I'm not sure it's
worth backpatching.

I'm on the fence.

- It's too much work to see what all failed across all the jobs. I've
experimented with generating a markdown summary across the jobs that ran
(basically a table that shows which steps succeeded and how many tests
failed/skipped/timed out, as well as the name of the first failed test).

It does require not entirely trivial changes. But it does make it faster to
grasp what's going on. It also perhaps is interesting for cfbot /
commitfest app, because it'd basically would include a summary of which
steps failed and how many tests passed/failed/... as an output of each job
and then the workflow.

That's a pretty substantial QOL improvement,

- Right now all logs get uploaded. That's quite the waste of space for
artifacts. Bilal has sent a patch: [2]

But this isn't a new problem, so perhaps it's ok to leave this for later?

- I comparison to cirrus-ci it's considerably more painful (and it wasn't
exactly pain-free on cirrus either) to access the logs of failed tasks. One
can't just link to the failure or such.

I have wondered about determining which test failed first, and uploading the
most crucial logs for that test separately, so one could at least look and
link to those without unpacking a .zip.

An argument against making that a hard requirement before backpatching is
that one needs to look at failures on master a lot more often than on the
back branches.

- A decent chunk of test time is spent setting up the containers (I've
optimized them a bit to reduce that already). Somehow docker is pretty slow
around container extraction. I had already split the containers into one
for docs and one for the rest, if we did that further, we could make startup
of e.g. sanitycheck (which has an outsized impact) a decent bit faster, but
we can't use the same container for e.g. linux-meson-32.

I think it may be smart to just add per-task tags for the containers. Then
we can have them initially be the same (by just pushing the same container
with different tags), which would allow us to adjust the containers contents
later, without needing to patch the workflow in the postgres repo.

I suspect we should do the tag aliases before backpatching, but I'm very
willing to be convinced otherwise.

Any opinions on the above? Any other points that we need to resolve before
backpatching?

Greetings,

Andres Freund

[1] https://postgr.es/m/CAN55FZ1BgsXSTzOpehnMa4NzWL8Aivsxx-di7-VT6bZ3j2Omow%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAN55FZ07AefTV_D2bCZae5jtQOQD1QByNe3FbXvM9Lq166c4og%40mail.gmail.com

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-10T20:29:33Z

Hi,

On 2026-06-10 10:55:21 -0400, Andres Freund wrote:
> On 2026-06-04 15:03:39 -0400, Andres Freund wrote:
> > I did futz around with ccache improvements for a bit. I think we're going to
> > need them, but they're complicated enough to do them separately.
>
> The ccache improvements have been committed since, in:
>   2026-06-08  f52c44ce48a  ci: Improve ccache handling
>
> Before we can backpatch the CI support I think we need to resolve a few more
> things:
>
> - re-enabling crash reporting for windows, Bilal sent a patch [1]
>
>   I think that's pretty much a must have, otherwise debuggin windows issues is
>   really hard.
>
>
> - Cold or inapplicable (e.g. due to a core header change) compiler warnings
>   task is very slow (35min).  I have a patch that I need to send out to
>   convert everything but the headercheck in compilerwarnings to meson, that
>   reduces the worst case build times considerably (primarily due to the cross
>   build being able to use precompiled headers)
>
>   I'll try to send that out later today.
>
>
>
> There's other issues, but I'm not sure we need to resolve them before
> backpatching:
>
> - Coverage for the BSDs - this is complicated enough that I'm not sure it's
>   worth backpatching.
>
>   I'm on the fence.
>
>
> - It's too much work to see what all failed across all the jobs. I've
>   experimented with generating a markdown summary across the jobs that ran
>   (basically a table that shows which steps succeeded and how many tests
>   failed/skipped/timed out, as well as the name of the first failed test).
>
>   It does require not entirely trivial changes. But it does make it faster to
>   grasp what's going on.  It also perhaps is interesting for cfbot /
>   commitfest app, because it'd basically would include a summary of which
>   steps failed and how many tests passed/failed/... as an output of each job
>   and then the workflow.
>
>   That's a pretty substantial QOL improvement,
>
>
> - Right now all logs get uploaded. That's quite the waste of space for
>   artifacts. Bilal has sent a patch: [2]
>
>   But this isn't a new problem, so perhaps it's ok to leave this for later?
>
>
> - I comparison to cirrus-ci it's considerably more painful (and it wasn't
>   exactly pain-free on cirrus either) to access the logs of failed tasks. One
>   can't just link to the failure or such.
>
>   I have wondered about determining which test failed first, and uploading the
>   most crucial logs for that test separately, so one could at least look and
>   link to those without unpacking a .zip.
>
>   An argument against making that a hard requirement before backpatching is
>   that one needs to look at failures on master a lot more often than on the
>   back branches.
>
>
> - A decent chunk of test time is spent setting up the containers (I've
>   optimized them a bit to reduce that already).  Somehow docker is pretty slow
>   around container extraction.  I had already split the containers into one
>   for docs and one for the rest, if we did that further, we could make startup
>   of e.g. sanitycheck (which has an outsized impact) a decent bit faster, but
>   we can't use the same container for e.g. linux-meson-32.
>
>   I think it may be smart to just add per-task tags for the containers. Then
>   we can have them initially be the same (by just pushing the same container
>   with different tags), which would allow us to adjust the containers contents
>   later, without needing to patch the workflow in the postgres repo.
>
>   I suspect we should do the tag aliases before backpatching, but I'm very
>   willing to be convinced otherwise.
>
>
> Any opinions on the above?  Any other points that we need to resolve before
> backpatching?

Two more things came to mind:

- caching the cpan install on windows, Jacob started working on that [1]

  I think this would be nice to have before backpatching, but that it's not
  required.


- What type of runner we use is not adjustable

  If one runs CI in a private repository, the performance is attrocious,
  because the github hosted runners for private repos are much weaker than the
  ones for public repos.  But as-is, there's no way to change the type of
  workers used without editing pg-ci.yml.  I think we should make it
  configurable via repository-level variables.

  One thing that's related to that is that we currently specify the
  concurrency in a variable, but that doesn't work well when increasing the
  size of runners.  I think for runners other than autoconf, we should just
  remove the explicit concurrency, as meson test and ninja just use the number
  of cores.  If we eventually find something where we need to limit / increase
  that, we can do so at that time.


Greetings,

Andres Freund


[1] https://postgr.es/m/CAOYmi%2BmC8_ZW3A1vGZHMDiW%2BvCMHQNga4jb9jawHn%3DLauwL6xQ%40mail.gmail.com

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-10T23:26:26Z

Hi,

On 2026-05-28 13:50:26 -0700, Jacob Champion wrote:
> > v3 is attached.
> 
> > +        uses: msys2/setup-msys2@v2
> 
> Should we pin this? It's the only third-party action we reference, and
> Scorecard [1] complains. (I'm not convinced its other complaints in
> this category are something we want to worry about, but this caught my
> eye.)

Isn't that a rather bogus complaint? After all, pacman is then used to install
a lot of stuff that's under control of the msys2/ org. And the github images
*also* install msys2 releases that are under control of the msys2/ org.  So
what increase in safety are we gaining by implementing this ourselves?

The reason I'm looking at it is that I was experimenting with using larger
runners for cfbot. Unfortunately they don't have a d:/ drive. Thus the mingw
task fails (there's also a sockdir issue, but that's trivial to fix).

I started to fix this by just installing msys ourselves [1], which also turns
out to be faster than moving the install, but then I considered that to be
somewhat too wheel-reinvent-y, compared to ust using msys2/setup-msys2.

Which lead me back here.

It turns out that using msys2/setup-msys2 might be tad slower than what I open
coded (i.e. copy-pasted from what we had for the image generation [3]). But it
does avoid downloading the packages over and over.

The only real alternative I see is to occasionally generate a downloadable
install in pg-vm-images. That'd probably be faster.

Greetings,

Andres Freund

[1] https://github.com/anarazel/postgres/actions/runs/27311452310/job/80682380845
[2] https://github.com/anarazel/postgres/actions/runs/27312189287/job/80685351695
[3] https://github.com/anarazel/pg-vm-images/blob/main/scripts/windows_install_mingw64.ps1

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-06-10T23:42:22Z

On Wed, Jun 10, 2026 at 4:26 PM Andres Freund <andres@anarazel.de> wrote:
> Isn't that a rather bogus complaint? After all, pacman is then used to install
> a lot of stuff that's under control of the msys2/ org. And the github images
> *also* install msys2 releases that are under control of the msys2/ org.  So
> what increase in safety are we gaining by implementing this ourselves?

1) It depends on whether you think it's as easy to poison upstream
MSYS servers as it is to poison a mutable GitHub tag.
2) I think we should *also* move away from live installs of the latest
versions of stuff, but that seems like a much heavier lift than just
pinning a tag, which is easy.

The goal isn't to completely avoid trusting any other software
organizations, but to avoid letting a GitHub supply chain attack
spread like wildfire.

> The reason I'm looking at it is that I was experimenting with using larger
> runners for cfbot. Unfortunately they don't have a d:/ drive. Thus the mingw
> task fails (there's also a sockdir issue, but that's trivial to fix).
>
> I started to fix this by just installing msys ourselves [1], which also turns
> out to be faster than moving the install, but then I considered that to be
> somewhat too wheel-reinvent-y, compared to ust using msys2/setup-msys2.
>
> Which lead me back here.

To clarify: I'm not against using setup-msys2 if you think it's of
good quality; I just thought the SHA should be pinned.

--Jacob

Re: Heads Up: cirrus-ci is shutting down June 1st

Jelte Fennema <postgres@jeltef.nl> — 2026-06-11T06:44:07Z

On Thu, 11 Jun 2026 at 01:42, Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
>
> On Wed, Jun 10, 2026 at 4:26 PM Andres Freund <andres@anarazel.de> wrote:
> > Isn't that a rather bogus complaint? After all, pacman is then used to install
> > a lot of stuff that's under control of the msys2/ org. And the github images
> > *also* install msys2 releases that are under control of the msys2/ org.  So
> > what increase in safety are we gaining by implementing this ourselves?
>
> 1) It depends on whether you think it's as easy to poison upstream
> MSYS servers as it is to poison a mutable GitHub tag.
> 2) I think we should *also* move away from live installs of the latest
> versions of stuff, but that seems like a much heavier lift than just
> pinning a tag, which is easy.
>
> The goal isn't to completely avoid trusting any other software
> organizations, but to avoid letting a GitHub supply chain attack
> spread like wildfire.

I don't really understand what actual problem is that you're trying to
protect against. i.e. what's the worst thing that a hostile takeover
of the msys github action (or any other action for that matter) can
result in?

We already allow anyone to run arbitrary CI on the postgresql-cfbot
repo by simply submitting a patch to the mainlinglist. This seems
fine, since we don't have any secrets associated with the repo.
Neither do we have any secrets on the postgres/postgres repo. Usually
what these attacks target secrets used to deploy or publish releases.
Our repos don't do any of that.

Re: Heads Up: cirrus-ci is shutting down June 1st

Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-11T09:04:29Z

On Wed, Jun 10, 2026 at 4:12 PM Andres Freund <andres@anarazel.de> wrote:

> Hi,
>
> On 2026-06-10 13:13:49 +0200, Jakub Wartak wrote:
> > On Tue, Jun 9, 2026 at 2:14 PM Andres Freund <andres@anarazel.de> wrote:
> > > On 2026-06-09 12:32:31 +0200, Jakub Wartak wrote:
> > > > And I've bumped TEST_JOBS 4->8 (even with 4 VCPUs), because my local
> > runs
> > > > showed in taskmgr that after quite some time we have ended up using
just
> > > > ~40% CPU (also with 4 VCPUs) while not doing I/O (this is somehow
> > contrary
> > > > to what Andres was stating earlier).
> > >
> > > FWIW, I only measured this for linux, not for windows. On linux it was
> > easy to
> > > do
> > >
> > > +  vmstat -y -n -w 1 > vmstat.log &
> > > +
> > > +  meson test ${{env.MTEST_ARGS}} --num-processes
> > ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}}
> > > +
> > > +  killall iostat vmstat || true
> > > +
> > > +  - name: Upload stats
> > > +uses: actions/upload-artifact@v7
> > > +with:
> > > +  path: |
> > > +iostat.log
> > > +vmstat.log
> > >
> > > Which showed that there is very little idle CPU other than during
first
> > few
> > > seconds and at the end.
> [..]
> > Attached are: patch how it was gathered, raw CSV data, and most
importantly
> > graph.
>
> Looking at the raw data, I think something must not be quite right. Note
how
> low the absolute read/write IO numbers are. Is it possible that that's
for the
> C:/ disk, but that we're doing IO on D:/?

It's "_Total", so should include everything. That would mean we are doing
I/O
somewhere.

> How exctly did you translate the csv data to %cpu utilization?

It's raw, and ""% Processor Time shows the total percentage of processor
utilization across all processes."

> > We were both right and wrong. It is either CPU bottleneck, but also
> > if the I/O is involved the CPU drops to <20% in case of runner #1 (same
> > happens with runner#2 but for short time of 2 mins). Pretty much had
> > similiar local Windows behavior.
>
> Kinda looks like what we might want is to increase the times / amounts
> equivalent to
>
/proc/sys/vm/{dirty_expire_centisecs,dirty_writeback_centisecs,dirty_background_ratio,dirty_ratio}
>
> But due to the issue mentioned above, I'm not sure we can conclude that
much
> yet.

I've searched GH issues and there are hundreths of people complaining
(and dozes of issues) that Windows is simply slower especially if I/O is
involved. I found this link [1] which is short and nice summary of those
issues. It has ready to use recipes, so I've used one for RAM disk
:rotfl: (on more serious note: I wanted to make it GH env "fast_noIO"=true
if not chaning anything to IO/fs), but ...:
- 'Test world' alone took 12min + 16min (so thats 2 runners each with
4VCPUs)
- and that was NOT much faster than normal than we had yesterday on D:\
  (and this was with R:\build on ramdisk, sic!)
- along the way I've captured metrics (attached) + graphs (see ..totals.jpg
  first), so we were are still IDLE on CPU when doing some I/O (_Total),
  but this is not I/O for ramdisk as one cannot have 10ms+ IO on ramdisk
  all the time, right? (not to mention those peaks to 400ms+)
- so I've collected more detailed per-disk IO data (_Total, but also "*")
  using attached patch
- and if you look at the second graph we are doing I/O on __very__ slow C:
  when are __slow__ during tests and that correlates with low CPU usage, so
  something is being used on C: is slowing us down to crawl
- I thrown 3rd run to collect per-process I/O into that CSV and later thrown
  that onto Claude to find cross-corellation between processes doing lots
of
  I/O, it confirmed, but didnt find anything specific:
    Your hypothesis is confirmed — but the cause isn't what you'd expect
    CPU and C: queue length are almost perfectly inversely correlated (r =
    ‑0.947).The clearest stretch is 08:12:09–08:13:09, where the C: queue
    pins at ~3.9–4.0 while CPU collapses toward 0%. There are weaker
    recurrences through ~08:18. But when I rank processes by actual I/O
    on C: during those exact windows, nobody is moving meaningful volume:
    [..slop mentioning everything a bit, including system-writeback and
    Azure throttling]
- btw: I've set TMP and TEMP to d:\wintmp and it did not help and I'm not
  Windows expert at all, but something there is borked and at least
  it's clear what (pagefile is already on D:)

To sum up, to me it looks like we are losing ~60..70% of compute on
Window due to that slow C: being issue, but I have to stop here.

> > IMHO *if* we want to push that faster it would make some sense to
eliminate
> > that I/O (but after observing that matrix split trich I'm not so sure
if it
> > is worth investing more into it). We seem to drop CPU use every time the
> > avg disk queue len >= 2.
>
> I'm not that concerned about the VS runtime right now, due to the split,
but
> mingw very frequently is the slowest task (with an empty / inapplicable
cache
> it's compilerwarnings, but I have some pending improvements for that, by
> converting it to meson the worst case time halves). We can't just split
all
> tasks, that uses too many of the available "job slots".
>
>
> > Or maybe offload that and ask GH folks to provide images with XFS and
ReFS
> > on D:\ by default instead ?
>
> I suspect that will be a very heavy lift.  That'd be a large change and
there
> are lot of users of this stuff.
>
> It's probably worth seeing what the times with a newer windows image are,
> before we do much more.

Simplest tweak s/windows-2022/windows-2025/ says:

  Run-time dependency openssl found: NO  (tried pkg-config and system)
  meson.build:1645:17: ERROR: C header 'openssl/ssl.h' not found

I remember we have installed openssl in previous CI patches, but not on what
is right now on master, dunno, I haven't pressed harder.

> > Alvaro had an idea here in [1] about instance reusing.
>
> We have the ability to run instances against a running cluster already,
but
> only use that in one place. I was wondering about a meson test "setup"
that
> will only run tests that can *not* be run against a running instance.
>
> With a bit of additional scripting (we need the ability to set
LD_LIBRARY_PATH
> in a cross platform, we have that in a bunch of places, just need to
expose
> it), that'd allow us to convert all the meson based tests to use the
running
> tests, and all the tests that don't support that, without duplication
between
> the runs.

it kind of sounds like black-magic wizardy to me, and that LD_LIBRARY_PATH
there, sorry, I'm not following , to override which libs? (not sure how
that's
supposed to work) :)

> It's not really a fair comparison (due to what's running concurrently),
but
> here's the time for a few tests in running and a dedicated cluster:
>
> 70/398 postgresql:bloom / bloom/regress   OK1.97s   1 subtests passed
> 6/88 postgresql:bloom-running / bloom-running/regress OK0.54s   1
subtests passed
>
> 68/398 postgresql:auto_explain / auto_explain/regress OK1.96s   2
subtests passed
> 5/88 postgresql:auto_explain-running / auto_explain-running/regress
OK0.33s   2 subtests passed
>
> 77/398 postgresql:cube / cube/regress OK2.27s   2 subtests passed
> 11/88 postgresql:cube-running / cube-running/regress  OK0.84s   2
subtests passed
>
> Clearly we could gain some if we we didn't run the tests that supported
> running against an existing cluster against separate clusters each.

... That's like 3x-4x :o

-J.

[1] -
https://chadgolden.com/blog/github-actions-hosted-windows-runners-slower-than-expected-ci-and-you

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-11T13:09:34Z

Hi,

On 2026-06-10 16:42:22 -0700, Jacob Champion wrote:
> On Wed, Jun 10, 2026 at 4:26 PM Andres Freund <andres@anarazel.de> wrote:
> > Isn't that a rather bogus complaint? After all, pacman is then used to install
> > a lot of stuff that's under control of the msys2/ org. And the github images
> > *also* install msys2 releases that are under control of the msys2/ org.  So
> > what increase in safety are we gaining by implementing this ourselves?
> 
> 1) It depends on whether you think it's as easy to poison upstream
> MSYS servers as it is to poison a mutable GitHub tag.

It's just as easy I think.


> 2) I think we should *also* move away from live installs of the latest
> versions of stuff, but that seems like a much heavier lift than just
> pinning a tag, which is easy.

I think you maybe understimate the noise of constant "bump version of xzy"
commits across N branches.


> The goal isn't to completely avoid trusting any other software
> organizations, but to avoid letting a GitHub supply chain attack
> spread like wildfire.

I guess I just don't see the supply chain danger here. This is for testing,
not for making releases. What's the threat model in which attacking postgres'
CI helps you spread the attack further?

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-06-11T16:08:49Z

On Thu, Jun 11, 2026 at 6:09 AM Andres Freund <andres@anarazel.de> wrote:
> > 1) It depends on whether you think it's as easy to poison upstream
> > MSYS servers as it is to poison a mutable GitHub tag.
>
> It's just as easy I think.

Okay. In that case it's probably not as useful to pin this, when
compared to the pg-vm-images alternative.

> I think you maybe understimate the noise of constant "bump version of xzy"
> commits across N branches.

Even if our MinGW setup action needs to be constantly on the
leading-edge, it released roughly once a quarter last year. (But I'd
say just update it when there's a security alert, or else switch to
pg-vm-images, to both speed things up and control the
reproducibility.)

> I guess I just don't see the supply chain danger here. This is for testing,
> not for making releases. What's the threat model in which attacking postgres'
> CI helps you spread the attack further?

Well, we went through a similar conversation upthread -- if no one
ever makes mistakes in the token permissions, and no one ever does
anything odd in a downstream fork, and GitHub doesn't turn out to have
a corner case that lets you escalate from a read-only token in an
unintuitive way, then I guess we're probably fine?

It's just a lot of 'if's, and the cost of pinning a single SHA really
didn't seem to outweigh all that to me, since GitHub has a bunch of
tools like dependabot that assist people who are pinning SHAs.

--Jacob

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-06-12T13:44:25Z

Hi,

On Wed, 10 Jun 2026 at 17:55, Andres Freund <andres@anarazel.de> wrote:
>
> Before we can backpatch the CI support I think we need to resolve a few more
> things:
>
> - re-enabling crash reporting for windows, Bilal sent a patch [1]
>
>   I think that's pretty much a must have, otherwise debuggin windows issues is
>   really hard.

That seems working based on my testing, just needs a review.


> - Cold or inapplicable (e.g. due to a core header change) compiler warnings
>   task is very slow (35min).  I have a patch that I need to send out to
>   convert everything but the headercheck in compilerwarnings to meson, that
>   reduces the worst case build times considerably (primarily due to the cross
>   build being able to use precompiled headers)
>
>   I'll try to send that out later today.

Nice. I encountered this problem a couple of times and they were frustrating.


> There's other issues, but I'm not sure we need to resolve them before
> backpatching:
>
> - Coverage for the BSDs - this is complicated enough that I'm not sure it's
>   worth backpatching.
>
>   I'm on the fence.

I think the current pg-ci.yml file is complicated enough. We can have
BSD support and their back-patches all together later.


> - It's too much work to see what all failed across all the jobs. I've
>   experimented with generating a markdown summary across the jobs that ran
>   (basically a table that shows which steps succeeded and how many tests
>   failed/skipped/timed out, as well as the name of the first failed test).
>
>   It does require not entirely trivial changes. But it does make it faster to
>   grasp what's going on.  It also perhaps is interesting for cfbot /
>   commitfest app, because it'd basically would include a summary of which
>   steps failed and how many tests passed/failed/... as an output of each job
>   and then the workflow.
>
>   That's a pretty substantial QOL improvement,

I agree. Also, since this doesn't change any behavior, it would be
(hopefully) easier to test & review this compared to other
improvements.


> - Right now all logs get uploaded. That's quite the waste of space for
>   artifacts. Bilal has sent a patch: [2]
>
>   But this isn't a new problem, so perhaps it's ok to leave this for later?

Definitely. Main purpose of this patch was reaching failed tests' logs
easily rather than saving a space. It is still a nice gain but this
can wait.


> - I comparison to cirrus-ci it's considerably more painful (and it wasn't
>   exactly pain-free on cirrus either) to access the logs of failed tasks. One
>   can't just link to the failure or such.
>
>   I have wondered about determining which test failed first, and uploading the
>   most crucial logs for that test separately, so one could at least look and
>   link to those without unpacking a .zip.

I have questions about this. If we do this for all jobs then we can
end up having just too many uploaded files to look at. In this case,
unpacking .zip would be easier to access the logs of failed tasks.


> - A decent chunk of test time is spent setting up the containers (I've
>   optimized them a bit to reduce that already).  Somehow docker is pretty slow
>   around container extraction.  I had already split the containers into one
>   for docs and one for the rest, if we did that further, we could make startup
>   of e.g. sanitycheck (which has an outsized impact) a decent bit faster, but
>   we can't use the same container for e.g. linux-meson-32.
>
>   I think it may be smart to just add per-task tags for the containers. Then
>   we can have them initially be the same (by just pushing the same container
>   with different tags), which would allow us to adjust the containers contents
>   later, without needing to patch the workflow in the postgres repo.
>
>   I suspect we should do the tag aliases before backpatching, but I'm very
>   willing to be convinced otherwise.

It would make sense to do this before backpatching as some of the
improvements can be done on the images themselves, which can speed up
the process.


--
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Andres Freund <andres@anarazel.de> — 2026-06-12T21:54:40Z

Hi,

Here's a series implementing a lot, but not all, that I brought up.  See the
commit messages for details.  One change that I hadn't mentioned on here is
that I increased the compiler optimizations for mingw, which improves the test
runtimes by a enough to even offset the cold-ccache compile times.

I for now used the setup-msys2 action, with a pin to the SHA to address
Jacob's concern (even though I still don't see what it buys us).

I did try creating an archive of an install, but that ends up with a good bit
of additional complexity, because the installations aren't relocatable (there
are header paths that are patched during install, apparently). And where the
install should be depends on the runner type, larger runners don't have D:,
but slower runners require using D:.  I think Bilal hacked further on my
experiments around this, but I don't quite know where that stands. It's
certainly not a trivial change.

I'm pretty agnostic whether we want 0005, I just included that because it
looks a bit nicer.

On 2026-06-12 16:44:25 +0300, Nazir Bilal Yavuz wrote:
> On Wed, 10 Jun 2026 at 17:55, Andres Freund <andres@anarazel.de> wrote:
> >
> > Before we can backpatch the CI support I think we need to resolve a few more
> > things:
> >
> > - re-enabling crash reporting for windows, Bilal sent a patch [1]
> >
> >   I think that's pretty much a must have, otherwise debuggin windows issues is
> >   really hard.
> 
> That seems working based on my testing, just needs a review.

I'll merge it, I think it's good to go.

> > - Cold or inapplicable (e.g. due to a core header change) compiler warnings
> >   task is very slow (35min).  I have a patch that I need to send out to
> >   convert everything but the headercheck in compilerwarnings to meson, that
> >   reduces the worst case build times considerably (primarily due to the cross
> >   build being able to use precompiled headers)
> >
> >   I'll try to send that out later today.
> 
> Nice. I encountered this problem a couple of times and they were frustrating.

Yea, it's definitely not great as-is.  It's a bit weird that with the patch
above we still use both meson and autoconf (for headerscheck), but I think
it's ok for now.

> > - I comparison to cirrus-ci it's considerably more painful (and it wasn't
> >   exactly pain-free on cirrus either) to access the logs of failed tasks. One
> >   can't just link to the failure or such.
> >
> >   I have wondered about determining which test failed first, and uploading the
> >   most crucial logs for that test separately, so one could at least look and
> >   link to those without unpacking a .zip.
> 
> I have questions about this. If we do this for all jobs then we can
> end up having just too many uploaded files to look at. In this case,
> unpacking .zip would be easier to access the logs of failed tasks.

I was thinking we'd do this only for the first failure...

> > - A decent chunk of test time is spent setting up the containers (I've
> >   optimized them a bit to reduce that already).  Somehow docker is pretty slow
> >   around container extraction.  I had already split the containers into one
> >   for docs and one for the rest, if we did that further, we could make startup
> >   of e.g. sanitycheck (which has an outsized impact) a decent bit faster, but
> >   we can't use the same container for e.g. linux-meson-32.
> >
> >   I think it may be smart to just add per-task tags for the containers. Then
> >   we can have them initially be the same (by just pushing the same container
> >   with different tags), which would allow us to adjust the containers contents
> >   later, without needing to patch the workflow in the postgres repo.
> >
> >   I suspect we should do the tag aliases before backpatching, but I'm very
> >   willing to be convinced otherwise.
> 
> It would make sense to do this before backpatching as some of the
> improvements can be done on the images themselves, which can speed up
> the process.

I did that on the container generation side and now attached a patch to use
them.  It does reduce the time a decent bit.

Greetings,

Andres Freund

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-06-15T23:20:02Z

On Fri, Jun 12, 2026 at 2:54 PM Andres Freund <andres@anarazel.de> wrote:
> I did try creating an archive of an install, but that ends up with a good bit
> of additional complexity, because the installations aren't relocatable (there
> are header paths that are patched during install, apparently). And where the
> install should be depends on the runner type, larger runners don't have D:,
> but slower runners require using D:.  I think Bilal hacked further on my
> experiments around this, but I don't quite know where that stands. It's
> certainly not a trivial change.

Huh. (How does the current robocopy relocation work, then?)

I'll see if the `inherit` path changes in 0003 have made it
easier/harder to get the CPAN cache working.

> I'm pretty agnostic whether we want 0005, I just included that because it
> looks a bit nicer.

It does look a lot nicer.

> > I have questions about this. If we do this for all jobs then we can
> > end up having just too many uploaded files to look at. In this case,
> > unpacking .zip would be easier to access the logs of failed tasks.
>
> I was thinking we'd do this only for the first failure...

+1, I like the idea.

--Jacob

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-06-17T11:21:32Z

Hi,

On Tue, 16 Jun 2026 at 02:20, Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
>
> On Fri, Jun 12, 2026 at 2:54 PM Andres Freund <andres@anarazel.de> wrote:
> > I did try creating an archive of an install, but that ends up with a good bit
> > of additional complexity, because the installations aren't relocatable (there
> > are header paths that are patched during install, apparently). And where the
> > install should be depends on the runner type, larger runners don't have D:,
> > but slower runners require using D:.  I think Bilal hacked further on my
> > experiments around this, but I don't quite know where that stands. It's
> > certainly not a trivial change.
>
> Huh. (How does the current robocopy relocation work, then?)

The current robocopy step happens before the packages are installed.
Therefore, it seems there is nothing problematic with the default MSYS
installation, but rather that packages installed later cause the
problem.


-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-06-18T11:37:13Z

Hi,

On Sat, 13 Jun 2026 at 00:54, Andres Freund <andres@anarazel.de> wrote:
>
> > > - I comparison to cirrus-ci it's considerably more painful (and it wasn't
> > >   exactly pain-free on cirrus either) to access the logs of failed tasks. One
> > >   can't just link to the failure or such.
> > >
> > >   I have wondered about determining which test failed first, and uploading the
> > >   most crucial logs for that test separately, so one could at least look and
> > >   link to those without unpacking a .zip.
> >
> > I have questions about this. If we do this for all jobs then we can
> > end up having just too many uploaded files to look at. In this case,
> > unpacking .zip would be easier to access the logs of failed tasks.
>
> I was thinking we'd do this only for the first failure...

I assume you mean first failure for the GHA run, not the first failure
per GHA job. Then, this makes sense. I just wonder how this can be
implemented.


I have only one comment on v13a-0002, I looked at the rest and all LGTM.

Subject: [PATCH v13a 2/9] ci: Use meson for most of CompilerWarnings, it's a
 lot faster

meson_common_features and linux_configure_features are not the same.
By using meson_common_features, we don't test:

    --with-gssapi
    --with-libcurl
    --with-llvm
    --with-pam
    --with-selinux
    --with-systemd
    --with-uuid=ossp

options. Is this a problem?


--
Regards,
Nazir Bilal Yavuz
Microsoft

Re: Heads Up: cirrus-ci is shutting down June 1st

Tatsuo Ishii <ishii@postgresql.org> — 2026-06-22T10:23:19Z

Hi,

Thank you for working on CFbot.

I would like to share a possible issue with Ci jobs found by Henson
and Matheus, particulary Linux Meson (64-bit).

Andres reported that while testing RPR patches, cfbot was failing, with crashes inside JIT.
https://www.postgresql.org/message-id/p7r5bekdbl2zcazid7agvfo2nfnq5bim2a5jkckqygld32n325%40fctfp6ou6qnb

Henson and Matheus analyzed the issue and came to the conclusion that
the crash is triggered by particular cflags passed to bitcode
generation, not the RPR patch itself.

In fact Henson succeeded in reproducing the crash with a trivial SQL like:
SELECT 1 AS result;
https://www.postgresql.org/message-id/CAAAe_zDtwoL8KaC_cpK4rU9jCrLxtGJ%3DTwLogdfH1PeE3_GT9Q%40mail.gmail.com

To fix the issue, can you please evaluate Matheus's patch?

His patch looks good to me.  I pushed Henson's regression test
addition to my GitHub PostgreSQL fork and confirmed Ci failed. Then I
pushed Matheus's patch and confirmed all Ci jobs succeeded.

https://www.postgresql.org/message-id/CAAAe_zDtwoL8KaC_cpK4rU9jCrLxtGJ%3DTwLogdfH1PeE3_GT9Q%40mail.gmail.com
(v2-0002-meson-strip-sanitizer.patch)

The patch looks good to me.
For your convenience, I include the patch in-line.
---- v2-0002-meson-strip-sanitizer.patch ---- 
>From b0311982e0ff06470d31c54b5d4822fd3c5e19d4 Mon Sep 17 00:00:00 2001
From: Henson Choi <assam258@gmail.com>
Date: Fri, 12 Jun 2026 09:15:10 +0900
Subject: [PATCH v2 2/3] Exclude sanitizer flags from LLVM JIT bitcode
 generation

The meson build passes c_args verbatim to the clang command that emits
the JIT bitcode.  Under -fsanitize=address the instrumentation ends up
in the bitcode and breaks the JIT: any JIT-compiled query crashes the
backend with SIGILL.  The autoconf build is unaffected, as it builds
BITCODE_CFLAGS from a whitelist that never includes CFLAGS.

Filter sanitizer flags out of c_args during bitcode generation.

Author: Matheus Alcantara <matheusssilv97@gmail.com>
Reviewer: Henson Choi <assam258@gmail.com>
---
 src/backend/jit/llvm/meson.build | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/src/backend/jit/llvm/meson.build b/src/backend/jit/llvm/meson.build
index 7df8453ad6f..1ebee3bdcaf 100644
--- a/src/backend/jit/llvm/meson.build
+++ b/src/backend/jit/llvm/meson.build
@@ -61,7 +61,23 @@ endif
 
 # XXX: Need to determine proper version of the function cflags for clang
 bitcode_cflags = ['-fno-strict-aliasing', '-fwrapv']
-bitcode_cflags += get_option('c_args')
+
+# Sanitizer instrumentation in the JIT bitcode corrupts the JIT code
+# generator: JIT-compiled queries crash with SIGILL.  Strip sanitizer flags
+# from c_args during bitcode generation, and warn when we do, since the
+# JIT-compiled code then runs without sanitizer coverage.
+bitcode_sanitize_stripped = false
+foreach cflag : get_option('c_args')
+  if cflag.contains('sanitize')
+    bitcode_sanitize_stripped = true
+foreach cflag : get_option('c_args')
+  if cflag.contains('sanitize')
+    bitcode_sanitize_stripped = true
+  else
+    bitcode_cflags += cflag
+  endif
+endforeach
+if bitcode_sanitize_stripped
+  warning('stripping sanitizer flags from LLVM JIT bitcode; JIT-compiled code will not be instrumented')
+endif
+
 bitcode_cflags += cppflags
 
 # XXX: Worth improving on the logic to find directories here
-- 
2.47.3
---- v2-0002-meson-strip-sanitizer.patch ----

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

Re: Heads Up: cirrus-ci is shutting down June 1st

Thomas Munro <thomas.munro@gmail.com> — 2026-06-23T03:10:23Z

On Mon, Jun 22, 2026 at 10:23 PM Tatsuo Ishii <ishii@postgresql.org> wrote:
> Andres reported that while testing RPR patches, cfbot was failing, with crashes inside JIT.
> https://www.postgresql.org/message-id/p7r5bekdbl2zcazid7agvfo2nfnq5bim2a5jkckqygld32n325%40fctfp6ou6qnb
>
> Henson and Matheus analyzed the issue and came to the conclusion that
> the crash is triggered by particular cflags passed to bitcode
> generation, not the RPR patch itself.
>
> In fact Henson succeeded in reproducing the crash with a trivial SQL like:
> SELECT 1 AS result;
> https://www.postgresql.org/message-id/CAAAe_zDtwoL8KaC_cpK4rU9jCrLxtGJ%3DTwLogdfH1PeE3_GT9Q%40mail.gmail.com
>
> To fix the issue, can you please evaluate Matheus's patch?

Will do, hopefully tomorrow.  I got a bit side-tracked and missed that
discussion.  Thanks!

Re: Heads Up: cirrus-ci is shutting down June 1st

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-06-23T22:54:17Z

On Mon, Jun 15, 2026 at 4:20 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> I'll see if the `inherit` path changes in 0003 have made it
> easier/harder to get the CPAN cache working.

Neither, it turns out, but I finally figured out what made the MinGW
`shell: bash` different from the MSVC `shell: bash`. The addition of
C:\msys64\usr\bin to the GITHUB_PATH appears to suppress the standard
PATH prefix, which is what I needed to be able to combine the two CPAN
implementations. That's attached (based on v13a).

Quick note on v13a-0004:

> +      - name: Install MSYS2
> +        uses: msys2/setup-msys2@66cd2cce69caa17b53920067426061ca1de3a884 # v2.31.1
> +        with:
> +          msystem: ${{env.MSYSTEM}}

Can we just get rid of the MSYSTEM envvar now? (And the
MSYS2_PATH_TYPE, if we replace it with `path-type: inherit` in the
action parameters?) See attached -0011.

--Jacob

Re: Heads Up: cirrus-ci is shutting down June 1st

Henson Choi <assam258@gmail.com> — 2026-06-29T00:25:51Z

Hi,

Since cfbot keeps coming up here, a quick bug report.

cfbot treats a .tgz attachment as a patch even when the archive has no
patch inside.  It then applies nothing, tests plain master, and reports
green -- so a real patch posted earlier drops out of testing and the CF
entry shows a false green.

Live example: CF 5802 (Unicode Normalization).  The last real patchset
is v11.  My next message attached coverage.tgz (a gcov HTML report, not
a patch); cfbot took that as the latest submission and has been testing
master ever since.  That's the reproducer -- and the reason 5802 looks
"stale/merged" on the dashboard when it isn't.

Suggested fix: only treat an attachment as a patch if it's a
.patch/.diff, or an archive containing one; otherwise ignore it and fall
back to the last attachment set that had patches.

Regards,
Henson

Re: Heads Up: cirrus-ci is shutting down June 1st

Thomas Munro <thomas.munro@gmail.com> — 2026-06-29T01:54:04Z

On Mon, Jun 29, 2026 at 12:26 PM Henson Choi <assam258@gmail.com> wrote:
> Since cfbot keeps coming up here, a quick bug report.
>
> cfbot treats a .tgz attachment as a patch even when the archive has no
> patch inside.  It then applies nothing, tests plain master, and reports
> green -- so a real patch posted earlier drops out of testing and the CF
> entry shows a false green.
>
> Live example: CF 5802 (Unicode Normalization).  The last real patchset
> is v11.  My next message attached coverage.tgz (a gcov HTML report, not
> a patch); cfbot took that as the latest submission and has been testing
> master ever since.  That's the reproducer -- and the reason 5802 looks
> "stale/merged" on the dashboard when it isn't.
>
> Suggested fix: only treat an attachment as a patch if it's a
> .patch/.diff, or an archive containing one; otherwise ignore it and fall
> back to the last attachment set that had patches.

Ack.  I will try to improve this...