Horton

Thread

Commits

  1. meson: add and use stamp files for generated headers

  1. meson: add and use stamp files for generated headers

    Andres Freund <andres@anarazel.de> — 2025-08-09T00:05:57Z

    Hi,

As $subject says, I'd like to introduce stamp files for generated headers. The
immediate motivation is that more than half of build.ninja just consists out
of repetitions of the list of generated headers (listed as order-only
dependencies). That makes invocations of ninja a tad slower and makes meson
take a bit longer to generate build.ninja.

Without the change:

    andres@awork3:/srv/dev/build/postgres/m-master-assert$ wc build.ninja
      12713  260280 7754708 build.ninja

    andres@awork3:/srv/dev/build/postgres/m-master-assert$ numactl --physcpubind 11 time ninja reconfigure
    ...
    3.48user 0.29system 0:03.84elapsed 97%CPU (0avgtext+0avgdata 87640maxresident)k
    0inputs+41520outputs (0major+71323minor)pagefaults 0swaps

    andres@awork3:/srv/dev/build/postgres/m-master-assert$ numactl --physcpubind 11 time ninja
    ninja: no work to do.
    0.06user 0.03system 0:00.10elapsed 98%CPU (0avgtext+0avgdata 22496maxresident)k
    0inputs+0outputs (0major+5216minor)pagefaults 0swaps


With the change:

    andres@awork3:/srv/dev/build/postgres/m-dev-assert$ wc build.ninja
      12733  143627 3376899 build.ninja

    andres@awork3:/srv/dev/build/postgres/m-dev-assert$ numactl --physcpubind 11 time ninja reconfigure
    ...
    3.14user 0.30system 0:03.53elapsed 97%CPU (0avgtext+0avgdata 74392maxresident)k
    0inputs+34576outputs (0major+66026minor)pagefaults 0swaps

    andres@awork3:/srv/dev/build/postgres/m-dev-assert$ numactl --physcpubind 11 time ninja
    ninja: no work to do.
    0.04user 0.02system 0:00.06elapsed 100%CPU (0avgtext+0avgdata 19332maxresident)k
    0inputs+0outputs (0major+4389minor)pagefaults 0swaps


My main motivation however is not those, in absolute terms, relatively small
speedups, but that I have some patches (LLVM bitcode generation, headerscheck
support, compile_commands.json targets for headers) that add more targets
depending on all those headers, and that the slowdown becomes more noticeable
with those. The improvements with just this patch seems a sufficient
improvement on its own, though - hence this thread.


It's a bit sad to use python to create the stamp files, it's more than 3x
slower than using 'touch' - but touch doesn't work on windows. It's also very
hard to imagine a case where creating stamp files is a significant bottleneck,
given that e.g. genbki.pl is somewhat expensive.

Greetings,

Andres Freund

  2. Re: meson: add and use stamp files for generated headers

    Nazir Bilal Yavuz <byavuz81@gmail.com> — 2025-08-11T11:40:40Z

    Hi,

Thank you for working on this!

On Sat, 9 Aug 2025 at 03:06, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> As $subject says, I'd like to introduce stamp files for generated headers. The
> immediate motivation is that more than half of build.ninja just consists out
> of repetitions of the list of generated headers (listed as order-only
> dependencies). That makes invocations of ninja a tad slower and makes meson
> take a bit longer to generate build.ninja.
>
> Without the change:
>
>     andres@awork3:/srv/dev/build/postgres/m-master-assert$ wc build.ninja
>       12713  260280 7754708 build.ninja
>
>     andres@awork3:/srv/dev/build/postgres/m-master-assert$ numactl --physcpubind 11 time ninja reconfigure
>     ...
>     3.48user 0.29system 0:03.84elapsed 97%CPU (0avgtext+0avgdata 87640maxresident)k
>     0inputs+41520outputs (0major+71323minor)pagefaults 0swaps
>
>     andres@awork3:/srv/dev/build/postgres/m-master-assert$ numactl --physcpubind 11 time ninja
>     ninja: no work to do.
>     0.06user 0.03system 0:00.10elapsed 98%CPU (0avgtext+0avgdata 22496maxresident)k
>     0inputs+0outputs (0major+5216minor)pagefaults 0swaps
>
>
> With the change:
>
>     andres@awork3:/srv/dev/build/postgres/m-dev-assert$ wc build.ninja
>       12733  143627 3376899 build.ninja
>
>     andres@awork3:/srv/dev/build/postgres/m-dev-assert$ numactl --physcpubind 11 time ninja reconfigure
>     ...
>     3.14user 0.30system 0:03.53elapsed 97%CPU (0avgtext+0avgdata 74392maxresident)k
>     0inputs+34576outputs (0major+66026minor)pagefaults 0swaps
>
>     andres@awork3:/srv/dev/build/postgres/m-dev-assert$ numactl --physcpubind 11 time ninja
>     ninja: no work to do.
>     0.04user 0.02system 0:00.06elapsed 100%CPU (0avgtext+0avgdata 19332maxresident)k
>     0inputs+0outputs (0major+4389minor)pagefaults 0swaps
>

I got similar results.

> My main motivation however is not those, in absolute terms, relatively small
> speedups, but that I have some patches (LLVM bitcode generation, headerscheck
> support, compile_commands.json targets for headers) that add more targets
> depending on all those headers, and that the slowdown becomes more noticeable
> with those. The improvements with just this patch seems a sufficient
> improvement on its own, though - hence this thread.
>
>
> It's a bit sad to use python to create the stamp files, it's more than 3x
> slower than using 'touch' - but touch doesn't work on windows. It's also very
> hard to imagine a case where creating stamp files is a significant bottleneck,
> given that e.g. genbki.pl is somewhat expensive.

I agree.

My only concern was losing this header dependency information in the
build.ninja file but build commands of these stamp files are in the
build.ninja and they include the list of headers, so it can be easily
found.

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

  3. Re: meson: add and use stamp files for generated headers

    Andres Freund <andres@anarazel.de> — 2025-08-11T19:27:51Z

    Hi,

On 2025-08-11 14:40:40 +0300, Nazir Bilal Yavuz wrote:
> Thank you for working on this!

Thanks for the review - pushed.

Greetings,

Andres Freund