Horton

Thread

Commits

  1. windows: Define WIN32_LEAN_AND_MEAN to make compilation faster.

  1. windows build slow due to windows.h includes

    Andres Freund <andres@anarazel.de> — 2021-09-21T19:30:35Z

    Hi,

For the AIO stuff I needed to build postgres for windows. And I was a bit
horrified by the long compile times. At first I was ready to blame the MS
compiler for being slow, until I noticed that using mingw gcc from linux to
cross compile to windows is also a *lot* slower than building for linux.

I found some blog-post-documented-only compiler flags [1], most importantly
/d1reportTime. Which shows that the include processing of postgres.h takes
0.6s [2]

Basically all the time in a debug windows build is spent parsing windows.h and
related headers. Argh.

The amount of stuff we include in win32_port.h and declare is pretty absurd
imo. There's really no need to expose the whole backend to all of it. Most of
it should just be needed in a few port/ files and a few select users.

But that's too much work for my taste. As it turns out there's a partial
solution to windows.h being just so damn big, the delightfully named
WIN32_LEAN_AND_MEAN.

This reduces the non-incremental buildtime in my 8 core windows VM from 187s to
140s. Cross compiling from linux it's
master:
real	0m53.807s
user	22m16.930s
sys	2m50.264s
WIN32_LEAN_AND_MEAN
real	0m32.956s
user	12m17.773s
sys	1m52.313s

Still far from !windows compile times, but still not a bad improvement.

Most of the compile time after this is still spent doing parsing /
preprocessing. I sidetracked myself into looking at precompiled headers, but
it's not trivial to do that right unfortunately.


I think it'd be good if win32_port.h were slimmed down, and more of its
contents were moved into fake "port/win32/$name-of-unix-header" style headers
or such.


Greetings,

Andres Freund

[1] https://aras-p.info/blog/2019/01/21/Another-cool-MSVC-flag-d1reportTime/

[2]

postgres.c
Include Headers:
        Count: 483
                c:\Users\anfreund\src\postgres\src\include\postgres.h: 0.561795s
                        c:\Users\anfreund\src\postgres\src\include\c.h: 0.556991s
                                c:\Users\anfreund\src\postgres\src\include\postgres_ext.h: 0.000488s
                                        c:\Users\anfreund\src\postgres\src\include\pg_config_ext.h: 0.000151s
                                c:\Users\anfreund\src\postgres\src\include\pg_config.h: 0.000551s
                                c:\Users\anfreund\src\postgres\src\include\pg_config_manual.h: 0.000286s
                                c:\Users\anfreund\src\postgres\src\include\pg_config_os.h: 0.014283s
                                        C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\crtdefs.h: 0.009727s
...
                                        c:\Users\anfreund\src\postgres\src\include\port\win32_port.h: 0.487469s
                                                C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\um\winsock2.h: 0.449373s
...
                                                        C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\um\windows.h: 0.439666s

  2. Re: windows build slow due to windows.h includes

    Andrew Dunstan <andrew@dunslane.net> — 2021-09-21T20:13:55Z

    On 9/21/21 3:30 PM, Andres Freund wrote:
> Hi,
>
> For the AIO stuff I needed to build postgres for windows. And I was a bit
> horrified by the long compile times. At first I was ready to blame the MS
> compiler for being slow, until I noticed that using mingw gcc from linux to
> cross compile to windows is also a *lot* slower than building for linux.
>
> I found some blog-post-documented-only compiler flags [1], most importantly
> /d1reportTime. Which shows that the include processing of postgres.h takes
> 0.6s [2]
>
> Basically all the time in a debug windows build is spent parsing windows.h and
> related headers. Argh.
>
> The amount of stuff we include in win32_port.h and declare is pretty absurd
> imo. There's really no need to expose the whole backend to all of it. Most of
> it should just be needed in a few port/ files and a few select users.
>
> But that's too much work for my taste. As it turns out there's a partial
> solution to windows.h being just so damn big, the delightfully named
> WIN32_LEAN_AND_MEAN.
>
> This reduces the non-incremental buildtime in my 8 core windows VM from 187s to
> 140s. Cross compiling from linux it's
> master:
> real	0m53.807s
> user	22m16.930s
> sys	2m50.264s
> WIN32_LEAN_AND_MEAN
> real	0m32.956s
> user	12m17.773s
> sys	1m52.313s


Nice!


I also see references to VC_EXTRALEAN which defines this and some other
stuff that might make things even faster.


Worth investigating.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  3. Re: windows build slow due to windows.h includes

    Andres Freund <andres@anarazel.de> — 2021-09-21T22:58:05Z

    Hi,

On 2021-09-21 16:13:55 -0400, Andrew Dunstan wrote:
> I also see references to VC_EXTRALEAN which defines this and some other
> stuff that might make things even faster.

I don't think that's relevant to "us", just mfc apps (which we gladly
aren't). From what I can see we'd have to actually clean up our includes to
not have windows.h everywhere or use precompiled headers to benefit further.

Greetings,

Andres Freund

  4. Re: windows build slow due to windows.h includes

    Ranier Vilela <ranier.vf@gmail.com> — 2021-09-21T23:26:36Z

    Em ter., 21 de set. de 2021 às 16:30, Andres Freund <andres@anarazel.de>
escreveu:

> Hi,
>
> For the AIO stuff I needed to build postgres for windows. And I was a bit
> horrified by the long compile times. At first I was ready to blame the MS
> compiler for being slow, until I noticed that using mingw gcc from linux to
> cross compile to windows is also a *lot* slower than building for linux.
>
> I found some blog-post-documented-only compiler flags [1], most importantly
> /d1reportTime. Which shows that the include processing of postgres.h takes
> 0.6s [2]
>
> Basically all the time in a debug windows build is spent parsing windows.h
> and
> related headers. Argh.
>
> The amount of stuff we include in win32_port.h and declare is pretty absurd
> imo. There's really no need to expose the whole backend to all of it. Most
> of
> it should just be needed in a few port/ files and a few select users.
>
> But that's too much work for my taste. As it turns out there's a partial
> solution to windows.h being just so damn big, the delightfully named
> WIN32_LEAN_AND_MEAN.
>
+1
But I did a quick dirty test here, and removed windows.h in win32_port.h,
and compiled normally with msvc 2019 (64 bit), would it work with mingw
cross compile?

regards,
Ranier Vilela

  5. Re: windows build slow due to windows.h includes

    Andres Freund <andres@anarazel.de> — 2021-09-21T23:56:43Z

    Hi,

On 2021-09-21 20:26:36 -0300, Ranier Vilela wrote:
> Em ter., 21 de set. de 2021 às 16:30, Andres Freund <andres@anarazel.de>
> escreveu:
> > But that's too much work for my taste. As it turns out there's a partial
> > solution to windows.h being just so damn big, the delightfully named
> > WIN32_LEAN_AND_MEAN.
> >
> +1
> But I did a quick dirty test here, and removed windows.h in win32_port.h,
> and compiled normally with msvc 2019 (64 bit), would it work with mingw
> cross compile?

That's likely only because winsock indirectly includes windows.h - because of
that it won't actually reduce compile time. And you can't remove the other
headers that indirectly include windows.h without causing compilation errors.

Greetings,

Andres Freund

  6. Re: windows build slow due to windows.h includes

    Noah Misch <noah@leadboat.com> — 2021-09-22T05:44:06Z

    On Tue, Sep 21, 2021 at 12:30:35PM -0700, Andres Freund wrote:
> solution to windows.h being just so damn big, the delightfully named
> WIN32_LEAN_AND_MEAN.
> 
> This reduces the non-incremental buildtime in my 8 core windows VM from 187s to
> 140s. Cross compiling from linux it's
> master:
> real	0m53.807s
> user	22m16.930s
> sys	2m50.264s
> WIN32_LEAN_AND_MEAN
> real	0m32.956s
> user	12m17.773s
> sys	1m52.313s

+1, great win for a one-liner.

  7. Re: windows build slow due to windows.h includes

    Juan José Santamaría Flecha <juanjo.santamaria@gmail.com> — 2021-09-22T07:06:03Z

    On Wed, Sep 22, 2021 at 1:56 AM Andres Freund <andres@anarazel.de> wrote:

> On 2021-09-21 20:26:36 -0300, Ranier Vilela wrote:
> > Em ter., 21 de set. de 2021 às 16:30, Andres Freund <andres@anarazel.de>
> > escreveu:
> > > But that's too much work for my taste. As it turns out there's a
> partial
> > > solution to windows.h being just so damn big, the delightfully named
> > > WIN32_LEAN_AND_MEAN.
> > >
> > +1
> > But I did a quick dirty test here, and removed windows.h in win32_port.h,
> > and compiled normally with msvc 2019 (64 bit), would it work with mingw
> > cross compile?
>
> That's likely only because winsock indirectly includes windows.h - because
> of
> that it won't actually reduce compile time. And you can't remove the other
> headers that indirectly include windows.h without causing compilation
> errors.
>
> You are right about winsock2.h including some parts of windows.h, please
see note in [1]. You could move the windows.h inclusion for clarity:

+ #ifndef WIN32_LEAN_AND_MEAN
+ #define WIN32_LEAN_AND_MEAN
+ #endif
+
+ #include <windows.h>
#include <winsock2.h>
#include <ws2tcpip.h>
- #include <windows.h>

[1]
https://docs.microsoft.com/en-us/windows/win32/winsock/creating-a-basic-winsock-application

Regards,

Juan José Santamaría Flecha

  8. Re: windows build slow due to windows.h includes

    Amit Kapila <amit.kapila16@gmail.com> — 2021-09-22T08:44:59Z

    On Wed, Sep 22, 2021 at 11:14 AM Noah Misch <noah@leadboat.com> wrote:
>
> On Tue, Sep 21, 2021 at 12:30:35PM -0700, Andres Freund wrote:
> > solution to windows.h being just so damn big, the delightfully named
> > WIN32_LEAN_AND_MEAN.
> >
> > This reduces the non-incremental buildtime in my 8 core windows VM from 187s to
> > 140s. Cross compiling from linux it's
> > master:
> > real  0m53.807s
> > user  22m16.930s
> > sys   2m50.264s
> > WIN32_LEAN_AND_MEAN
> > real  0m32.956s
> > user  12m17.773s
> > sys   1m52.313s
>
> +1, great win for a one-liner.
>

+1. It reduced the build time of Postgres from "Time Elapsed
00:01:57.60" to "Time Elapsed 00:01:38.11" in my Windows env. (Win 10,
MSVC 2019).

-- 
With Regards,
Amit Kapila.

  9. Re: windows build slow due to windows.h includes

    Michael Paquier <michael@paquier.xyz> — 2021-09-23T11:51:38Z

    On Wed, Sep 22, 2021 at 02:14:59PM +0530, Amit Kapila wrote:
> On Wed, Sep 22, 2021 at 11:14 AM Noah Misch <noah@leadboat.com> wrote:
> > +1, great win for a one-liner.
> 
> +1. It reduced the build time of Postgres from "Time Elapsed
> 00:01:57.60" to "Time Elapsed 00:01:38.11" in my Windows env. (Win 10,
> MSVC 2019).

That's nice.  Great find!
--
Michael