Horton
Menu

Re: CI and test improvements

Justin Pryzby <pryzby@telsasoft.com>

View thread
From: Justin Pryzby <pryzby@telsasoft.com>
To: Andres Freund <andres@anarazel.de>
Cc: Andrew Dunstan <andrew@dunslane.net>, Thomas Munro <thomas.munro@gmail.com>, pgsql-hackers@lists.postgresql.org, Noah Misch <noah@leadboat.com>, Michael Paquier <michael@paquier.xyz>, Anastasia Lubennikova <lubennikovaav@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>, Robert Haas <robertmhaas@gmail.com>, Melanie Plageman <melanieplageman@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, Daniel Gustafsson <daniel@yesql.se>
Date: 2022-10-02T21:35:06Z
Lists: pgsql-hackers 
On Sun, Oct 02, 2022 at 01:52:01PM -0700, Andres Freund wrote:
> Hi,
> 
> On 2022-10-01 18:36:41 -0700, Andres Freund wrote:
> > I am wondering if we should instead introduce a new "quickcheck" task that
> > just compiles and runs maybe one test and have *all* other tests depend on
> > that.  Wasting a precious available windows instance to just fail to build or
> > immediately fail during tests doesn't really make sense.

> With a primed cache this takes ~32s, not too bad imo. 12s of that is
> cloning the repo.

Maybe - that would avoid waiting 4 minutes for a windows instance to
start in the (hopefully atypical) case of a patch that fails in 1-2
minutes under linux/freebsd.

If the patch were completely broken, the windows task would take ~4min
to start, plus up to ~4min before failing to compile or failing an early
test.  6-8 minutes isn't nothing, but doesn't seem worth the added
complexity.

Also, this would mean that in the common case, the slowest task would be
delayed until after the SanityCheck task instance starts, compiles, and
runs some test :( Your best case is 32sec, but I doubt that's going to
be typical.

I was thinking about the idea of cfbot handling "tasks" separately,
similar to what it used to do with travis/appveyor.  The logic for
"windows tasks are only run if linux passes tests" could live there.
That could also be useful if there's ever the possibility of running an
additional OS on another CI provider, or if another provider can run
windows tasks faster, or if we need to reduce our load/dependency on
cirrus.  I realized that goes backwards in some ways to the direction
we've gone with cirrus, and I'm not sure how exactly it would do that (I
suppose it might add ci-os-only tags to its commit message).

> +    # no options enabled, should be small
> +    CCACHE_MAXSIZE: "150M"

Actually, tasks can share caches if the "cache key" is set.

If there was a separate "Sanity" task, I think it should use whatever
flags linux (or freebsd) use to avoid doing two compilations (with lots
of cache misses for patches which modify *.h files, which would then
happen twice, in serial).

> +  # use always: to continue after failures. Task that did not run count as a
> +  # success, so we need to recheck SanityChecks's condition here ...

> -  # task that did not run, count as a success, so we need to recheck Linux'
> -  # condition here ...

Another/better justification/description is that "cirrus warns if the
depending task has different only_if conditions than the dependant task".

-- 
Justin

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. ci: Add test coverage of different pg_upgrade modes

  2. seg: Add test "security" in meson.build

  3. cirrus/freebsd: define ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS

  4. cirrus/ccache: Use G rather than GB suffix

  5. ci: Upgrade macOS version from 12 to 13.

  6. meson: Add two missing regress tests

  7. Push lpp variable closer to usage in heapgetpage()

  8. ci: Change macOS builds from Intel to ARM.

  9. ci: Introduce SanityCheck task that other tasks depend on

  10. ci: Use -fsanitize=undefined,alignment,address in linux tasks

  11. ci: Clean up pre-meson cruft in windows task

  12. meson: Mark PROVE as not required

  13. ci: enable various runtime checks on FreeBSD and macOS