Horton

Thread

Commits

  1. ci: Use a RAM disk and more CPUs on FreeBSD.

  1. CI speed improvements for FreeBSD

    Thomas Munro <thomas.munro@gmail.com> — 2023-08-27T22:29:39Z

    Hi,

Here are a couple of changes that got FreeBSD down to 4:29 total, 2:40
in test_world in my last run (over 2x speedup), using a RAM disk
backed by a swap partition, and more CPUs.  It's still a regular UFS
file system but FreeBSD is not as good at avoiding I/O around short
lived files and directories as Linux: it can get hung up on a bunch of
synchronous I/O, and also flushes disk caches for those writes,
without an off switch.

I don't know about Windows, but I suspect the same applies there, ie
synchronous I/O blocking system calls around our blizzard of file
creations and unlinks.  Anyone know how to try it?

  2. Re: CI speed improvements for FreeBSD

    Thomas Munro <thomas.munro@gmail.com> — 2023-08-27T23:24:39Z

    And after adding this to the commitfest, here's the first cfbot run.
The gain was due to "test_world" which shows a greater-than-2x speedup
(~4:30 -> ~2:08) from 2x CPUs.  That is nice for humans who want the
answer as soon as possible, but note that the resource usage cost
might go up because of the non-parallel parts now wasting more idle
CPUs: git clone, meson configure etc (as they do on every platform).

https://cirrus-ci.com/build/6060109692928000

  3. Re: CI speed improvements for FreeBSD

    Maxim Orlov <orlovmg@gmail.com> — 2024-03-12T15:50:06Z

    Hi!

I looked at the changes and I liked them.  Here are my thoughts:

0001:
1. I think, this is a good idea to use RAM.  Since, it's still a UFS, and
we lose nothing in terms of testing, but win in speed significantly.
2. Change from "swapoff -a || true" to "swapoff -a" is legit in my view,
since it's better to explicitly fail than silent any possible problem.
3. Man says that lowercase suffixes should be used for the mdconfig. But in
fact, you can use either lowercase or an appercase. Yep, it's in
   the mdconfig.c: "else if (*p == 'g' || *p == 'G')".

0002:
1. The resource usage should be a bit higher, this is for sure.  But, if
I'm not missing something, not drastically. Anyway, I do not know
   how to measure this increase to get concrete values.
2. And think of a potential benefits of increasing the number of test jobs:
more concurrent processes, more interactions, better test coverage.

Here are my runs:
FreeBSD @master
https://cirrus-ci.com/task/4934701194936320
Run test_world 05:56

FreeBSD @master + 0001
https://cirrus-ci.com/task/5921385306914816
Run test_world 05:06

FreeBSD @master + 0001, + 0002
https://cirrus-ci.com/task/5635288945393664
Run test_world 02:20

For comparison
Debian @master
https://cirrus-ci.com/task/5143705577848832
Run test_world 02:38

In the overall, I consider this changes useful.  CI run faster, with better
test coverage in exchange for presumably slight increase
in resource usage, but I don't think this increase should be significant.

-- 
Best regards,
Maxim Orlov.

  4. Re: CI speed improvements for FreeBSD

    Thomas Munro <thomas.munro@gmail.com> — 2024-03-13T02:03:01Z

    On Wed, Mar 13, 2024 at 4:50 AM Maxim Orlov <orlovmg@gmail.com> wrote:
> I looked at the changes and I liked them.  Here are my thoughts:

Thanks for looking!  Pushed.