Horton

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. meson: Add dependency lookups via names used by cmake

  2. meson: Add support for detecting gss without pkg-config

  3. meson: Add support for detecting ossp-uuid without pkg-config

  1. Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-06-18T13:53:53Z

    Hi

Further to my previous report [1] about zlib detection not working with
Meson on Windows, I found it's similarly or entirely broken for the
majority of other dependencies, none of which are tested on the buildfarm
as far as I can see.

For convenience, I've put together a number of Github actions [2] that show
how to build the various dependencies on Windows, in the most
standard/recommended way I can find for each. Another action combines these
into a single downloadable archive that people can test with, and another
one uses that archive to build PostgreSQL 12 through 16, all successfully.

You can see build logs, and download the various builds/artefacts from the
Github Workflow pages.

My next task was to extend that to support PostgreSQL 17 and beyond, which
is where I started to run into problems. I've attempted builds using Meson
with each of the dependencies defined in the old-style config.pl, both with
and without modifying the INCLUDE/LIBS envvars to include the directories
for the dependencies (as was found to work in the previous discussion re
zlib):

Will not successfully configure at all:

meson setup --auto-features=disabled
-Dextra_include_dirs=C:\build64\include
-Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dgssapi=enabled
build-gssapi

meson setup --auto-features=disabled
-Dextra_include_dirs=C:\build64\include
-Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dicu=enabled
build-icu

meson setup --auto-features=disabled
-Dextra_include_dirs=C:\build64\include
-Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dlibxml=enabled
build-libxml

meson setup --auto-features=disabled
-Dextra_include_dirs=C:\build64\include
-Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dlz4=enabled
build-lz4

meson setup --auto-features=disabled
-Dextra_include_dirs=C:\build64\include
-Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dnls=enabled
build-nls

meson setup --auto-features=disabled
-Dextra_include_dirs=C:\build64\include
-Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Duuid=ossp
build-uuid

meson setup --auto-features=disabled
-Dextra_include_dirs=C:\build64\include
-Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dzstd=enabled
build-zstd

Configured with modified LIBS/INCLUDE:

meson setup --auto-features=disabled
-Dextra_include_dirs=C:\build64\include
-Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dlibxslt=enabled
build-libxslt

meson setup --auto-features=disabled
-Dextra_include_dirs=C:\build64\include
-Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dssl=openssl
build-openssl

meson setup --auto-features=disabled
-Dextra_include_dirs=C:\build64\include
-Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dzlib=enabled
build-zlib

I think it's important to note that Meson largely seems to want to use
pkgconfig and cmake to find dependencies. pkgconfig isn't really a thing on
Windows (it is available, but isn't commonly used), and even cmake would
typically rely on finding things in either known installation directories
or through lib/include vars. There really aren't standard directories like
/usr/lib or /usr/include as we find on unixes, or pkgconfig files for
everything.

For the EDB installers, the team has hand-crafted pkgconfig files for
everything, which is clearly not a proper solution.

I can provide logs and run tests if anyone wants me to do so. Testing so
far has been with the Ninja backend, in a VS2022 x86_64 native environment.

[1]
https://www.postgresql.org/message-id/CA+OCxozrPZx57ue8rmhq6CD1Jic5uqKh80=vTpZurSKESn-dkw@mail.gmail.com
[2] https://github.com/dpage/winpgbuild/actions

-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

  2. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-06-18T14:38:50Z

    Hi,

On 2024-06-18 14:53:53 +0100, Dave Page wrote:
> My next task was to extend that to support PostgreSQL 17 and beyond, which
> is where I started to run into problems. I've attempted builds using Meson
> with each of the dependencies defined in the old-style config.pl, both with
> and without modifying the INCLUDE/LIBS envvars to include the directories
> for the dependencies (as was found to work in the previous discussion re
> zlib):
> 
> Will not successfully configure at all:

Do you have logs for those failures?


> meson setup --auto-features=disabled
> -Dextra_include_dirs=C:\build64\include
> -Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dgssapi=enabled
> build-gssapi

> meson setup --auto-features=disabled
> -Dextra_include_dirs=C:\build64\include
> -Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dicu=enabled
> build-icu
> 
> meson setup --auto-features=disabled
> -Dextra_include_dirs=C:\build64\include
> -Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dlibxml=enabled
> build-libxml
> 
> meson setup --auto-features=disabled
> -Dextra_include_dirs=C:\build64\include
> -Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dlz4=enabled
> build-lz4
> 
> meson setup --auto-features=disabled
> -Dextra_include_dirs=C:\build64\include
> -Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dnls=enabled
> build-nls
> 
> meson setup --auto-features=disabled
> -Dextra_include_dirs=C:\build64\include
> -Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Duuid=ossp
> build-uuid
> 
> meson setup --auto-features=disabled
> -Dextra_include_dirs=C:\build64\include
> -Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dzstd=enabled
> build-zstd
> 
> Configured with modified LIBS/INCLUDE:
> 
> meson setup --auto-features=disabled
> -Dextra_include_dirs=C:\build64\include
> -Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dlibxslt=enabled
> build-libxslt
> 
> meson setup --auto-features=disabled
> -Dextra_include_dirs=C:\build64\include
> -Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dssl=openssl
> build-openssl
> 
> meson setup --auto-features=disabled
> -Dextra_include_dirs=C:\build64\include
> -Dextra_lib_dirs=C:\build64\lib;C:\build64\lib64 --wipe -Dzlib=enabled
> build-zlib
> 
> I think it's important to note that Meson largely seems to want to use
> pkgconfig and cmake to find dependencies. pkgconfig isn't really a thing on
> Windows (it is available, but isn't commonly used), and even cmake would
> typically rely on finding things in either known installation directories
> or through lib/include vars.

I am not really following what you mean with the cmake bit here?

You can configure additional places to search for cmake files with
meson setup --cmake-prefix-path=...


> There really aren't standard directories like
> /usr/lib or /usr/include as we find on unixes, or pkgconfig files for
> everything.

Yes, and?

Greetings,

Andres Freund

  3. Re: Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-06-18T14:54:27Z

    Hi

On Tue, 18 Jun 2024 at 15:38, Andres Freund <andres@anarazel.de> wrote:

> Hi,
>
> On 2024-06-18 14:53:53 +0100, Dave Page wrote:
> > My next task was to extend that to support PostgreSQL 17 and beyond,
> which
> > is where I started to run into problems. I've attempted builds using
> Meson
> > with each of the dependencies defined in the old-style config.pl, both
> with
> > and without modifying the INCLUDE/LIBS envvars to include the directories
> > for the dependencies (as was found to work in the previous discussion re
> > zlib):
> >
> > Will not successfully configure at all:
>
> Do you have logs for those failures?
>

Sure - https://developer.pgadmin.org/~dpage/build-logs.zip. Those are all
without any modifications to %LIB% or %INCLUDE%.


> I think it's important to note that Meson largely seems to want to use
> > pkgconfig and cmake to find dependencies. pkgconfig isn't really a thing
> on
> > Windows (it is available, but isn't commonly used), and even cmake would
> > typically rely on finding things in either known installation directories
> > or through lib/include vars.
>
> I am not really following what you mean with the cmake bit here?
>
> You can configure additional places to search for cmake files with
> meson setup --cmake-prefix-path=...
>

None of the dependencies include cmake files for distribution on Windows,
so there are no additional files to tell meson to search for. The same
applies to pkgconfig files, which is why the EDB team had to manually craft
them.


>
>
> > There really aren't standard directories like
> > /usr/lib or /usr/include as we find on unixes, or pkgconfig files for
> > everything.
>
> Yes, and?
>

And that's why we really need to be able to locate headers and libraries
easily by passing paths to meson, as we can't rely on pkgconfig, cmake, or
things being in some standard directory on Windows.

Thanks.

-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

  4. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-06-18T16:08:45Z

    Hi,


On 2024-06-18 15:54:27 +0100, Dave Page wrote:
> On Tue, 18 Jun 2024 at 15:38, Andres Freund <andres@anarazel.de> wrote:
> > Do you have logs for those failures?
> >
>
> Sure - https://developer.pgadmin.org/~dpage/build-logs.zip. Those are all
> without any modifications to %LIB% or %INCLUDE%.

Thanks.


> > I think it's important to note that Meson largely seems to want to use
> > > pkgconfig and cmake to find dependencies. pkgconfig isn't really a thing
> > on
> > > Windows (it is available, but isn't commonly used), and even cmake would
> > > typically rely on finding things in either known installation directories
> > > or through lib/include vars.
> >
> > I am not really following what you mean with the cmake bit here?
> >
> > You can configure additional places to search for cmake files with
> > meson setup --cmake-prefix-path=...
> >
>
> None of the dependencies include cmake files for distribution on Windows,
> so there are no additional files to tell meson to search for. The same
> applies to pkgconfig files, which is why the EDB team had to manually craft
> them.

Many of them do include at least cmake files on windows if you build them
though?


Btw, I've been working with Bilal to add a few of the dependencies to the CI
images so we can test those automatically. Using vcpkg. We got that nearly
working, but he's on vacation this week...  That does ensure both cmake and
.pc files are generated, fwiw.

Currently builds gettext, icu, libxml2, libxslt, lz4, openssl, pkgconf,
python3, tcl, zlib, zstd.



I'm *NOT* sure that vcpkg is the way to go, fwiw. It does seem advantageous to
use one of the toolkits thats commonly built for building dependencies on
windows, which seems to mean vcpkg or conan.


> And that's why we really need to be able to locate headers and libraries
> easily by passing paths to meson, as we can't rely on pkgconfig, cmake, or
> things being in some standard directory on Windows.

Except that that often causes hard to diagnose breakages, because that doesn't
allow including the necessary compiler/linker flags [2].  It's a bad model, we shouldn't
perpetuate it.  If we want to forever make windows a complicated annoying
stepchild, that's the way to go.

FWIW, at least libzstd, libxml [3], lz4, zlib can generate cmake dependency
files on windows in their upstream code.

I'm *not* against adding "hardcoded" dependency lookup stuff for libraries
where other approaches aren't feasible, I just don't think it's a good idea to
add fragile stuff that will barely be tested, when not necessary.

Greetings,

Andres Freund


[1] Here's a build of PG with the dependencies installed, builds
    https://cirrus-ci.com/task/4953968097361920

[2] E.g.
    https://github.com/postgres/postgres/blob/REL_16_STABLE/src/tools/msvc/Mkvcbuild.pm#L600
    https://github.com/postgres/postgres/blob/REL_16_STABLE/src/tools/msvc/Solution.pm#L1039

[3] Actually, at least your libxml build actually *did* include both .pc and
    cmake files. So just pointing to the relevant path would do the trick.

  5. Re: Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-06-19T13:47:50Z

    Hi

On Tue, 18 Jun 2024 at 17:08, Andres Freund <andres@anarazel.de> wrote:

> > None of the dependencies include cmake files for distribution on Windows,
> > so there are no additional files to tell meson to search for. The same
> > applies to pkgconfig files, which is why the EDB team had to manually
> craft
> > them.
>
> Many of them do include at least cmake files on windows if you build them
> though?
>

The only one that does is libxml2 as far as I can see. And that doesn't
seem to work even if I use --cmake-prefix-path= as you suggested:

C:\Users\dpage\git\postgresql>meson setup --auto-features=disabled --wipe
-Dlibxml=enabled --cmake-prefix-path=C:\build64\lib\cmake\libxml2-2.11.8
build-libxml
The Meson build system
Version: 1.4.0
Source dir: C:\Users\dpage\git\postgresql
Build dir: C:\Users\dpage\git\postgresql\build-libxml
Build type: native build
Project name: postgresql
Project version: 17beta1
C compiler for the host machine: cl (msvc 19.39.33523 "Microsoft (R) C/C++
Optimizing Compiler Version 19.39.33523 for x64")
C linker for the host machine: link link 14.39.33523.0
Host machine cpu family: x86_64
Host machine cpu: x86_64
Run-time dependency threads found: YES
Library ws2_32 found: YES
Library secur32 found: YES
Program perl found: YES (C:\Strawberry\perl\bin\perl.EXE)
Program python found: YES (C:\Python312\python.EXE)
Program win_flex found: YES 2.6.4 2.6.4
(C:\ProgramData\chocolatey\bin\win_flex.EXE)
Program win_bison found: YES 3.7.4 3.7.4
(C:\ProgramData\chocolatey\bin\win_bison.EXE)
Program sed found: YES (C:\ProgramData\chocolatey\bin\sed.EXE)
Program prove found: YES (C:\Strawberry\perl\bin\prove.BAT)
Program tar found: YES (C:\Windows\system32\tar.EXE)
Program gzip found: YES (C:\ProgramData\chocolatey\bin\gzip.EXE)
Program lz4 found: NO
Program openssl found: YES (C:\build64\bin\openssl.EXE)
Program zstd found: NO
Program dtrace skipped: feature dtrace disabled
Program config/missing found: YES (sh
C:\Users\dpage\git\postgresql\config/missing)
Program cp found: YES (C:\Program Files (x86)\GnuWin32\bin\cp.EXE)
Program xmllint found: YES (C:\build64\bin\xmllint.EXE)
Program xsltproc found: YES (C:\build64\bin\xsltproc.EXE)
Program wget found: YES (C:\ProgramData\chocolatey\bin\wget.EXE)
Program C:\Python312\Scripts\meson found: YES
(C:\Python312\Scripts\meson.exe)
Check usable header "bsd_auth.h" skipped: feature bsd_auth disabled
Check usable header "dns_sd.h" skipped: feature bonjour disabled
Compiler for language cpp skipped: feature llvm disabled
Found pkg-config: YES (C:\ProgramData\chocolatey\bin\pkg-config.EXE) 0.28
Found CMake: C:\Program Files\Microsoft Visual
Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.EXE
(3.28.0)
Run-time dependency libxml-2.0 found: NO (tried pkgconfig and cmake)

meson.build:796:11: ERROR: Dependency "libxml-2.0" not found, tried
pkgconfig and cmake

A full log can be found at
C:\Users\dpage\git\postgresql\build-libxml\meson-logs\meson-log.txt


>
>
> Btw, I've been working with Bilal to add a few of the dependencies to the
> CI
> images so we can test those automatically. Using vcpkg. We got that nearly
> working, but he's on vacation this week...  That does ensure both cmake and
> .pc files are generated, fwiw.
>
> Currently builds gettext, icu, libxml2, libxslt, lz4, openssl, pkgconf,
> python3, tcl, zlib, zstd.


That appears to be using Mingw/Msys, which is quite different from a VC++
build, in part because it's a full environment with its own package manager
and packages that people have put a lot of effort into making work as they
do on unix.


> I'm *NOT* sure that vcpkg is the way to go, fwiw. It does seem
> advantageous to
> use one of the toolkits thats commonly built for building dependencies on
> windows, which seems to mean vcpkg or conan.
>

I don't think requiring or expecting vcpkg or conan is reasonable at all,
for a number of reasons:

- Neither supports all the dependencies at present.
- There are real supply chain verification concerns for vendors.
- That would be a huge change from what we've required in the last 19
years, with no deprecation notices or warnings for packagers etc.


> > And that's why we really need to be able to locate headers and libraries
> > easily by passing paths to meson, as we can't rely on pkgconfig, cmake,
> or
> > things being in some standard directory on Windows.
>
> Except that that often causes hard to diagnose breakages, because that
> doesn't
> allow including the necessary compiler/linker flags [2].  It's a bad
> model, we shouldn't
> perpetuate it.  If we want to forever make windows a complicated annoying
> stepchild, that's the way to go.
>

That is a good point, though I suspect it wouldn't solve your second
example of the Kerberos libraries, as you'll get both 32 and 64 bit libs if
you follow their standard process for building on Windows so you still need
to have code to pick the right ones.


>
> FWIW, at least libzstd, libxml [3], lz4, zlib can generate cmake dependency
> files on windows in their upstream code.
>

In the case of zstd, it does not if you build with VC++, the Makefile, or
Meson, at least in my testing. It looks like it would if you built it
with cmake, but I couldn't get that to work in 10 minutes or so of messing
around. And that's a perfect example of what I'm bleating about - there are
often many ways of building things on Windows and there are definitely many
ways of getting things on Windows, and they're not all equal. We've either
got to be extremely prescriptive in our docs, telling people precisely what
they need to download for each dependency, or how to build it themselves in
the way that will work with PostgreSQL, or the build system needs to be
flexible enough to handle different dependency variations, as the old VC++
build system was.


>
> I'm *not* against adding "hardcoded" dependency lookup stuff for libraries
> where other approaches aren't feasible, I just don't think it's a good
> idea to
> add fragile stuff that will barely be tested, when not necessary.
>
> Greetings,
>
> Andres Freund
>
>
> [1] Here's a build of PG with the dependencies installed, builds
>     https://cirrus-ci.com/task/4953968097361920
>
> [2] E.g.
>
> https://github.com/postgres/postgres/blob/REL_16_STABLE/src/tools/msvc/Mkvcbuild.pm#L600
>
> https://github.com/postgres/postgres/blob/REL_16_STABLE/src/tools/msvc/Solution.pm#L1039
>
> [3] Actually, at least your libxml build actually *did* include both .pc
> and
>     cmake files. So just pointing to the relevant path would do the trick.
>


-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

  6. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-06-20T20:58:49Z

    Hi,

On 2024-06-19 14:47:50 +0100, Dave Page wrote:

> > I'm *NOT* sure that vcpkg is the way to go, fwiw. It does seem
> > advantageous to
> > use one of the toolkits thats commonly built for building dependencies on
> > windows, which seems to mean vcpkg or conan.
> >
>
> I don't think requiring or expecting vcpkg or conan is reasonable at all,
> for a number of reasons:
>
> - Neither supports all the dependencies at present.
> - There are real supply chain verification concerns for vendors.
> - That would be a huge change from what we've required in the last 19
> years, with no deprecation notices or warnings for packagers etc.

I don't think we should hard-require one specifically. I do think it'd be good
if we provided an easy recipe for dependencies to be installed though.


I think such flexibility acually means it becomes *more* important to abstract
away some of the concrete ways of using the various dependencies. It doesn't
make sense for postgres to understand the internals of each dependency on all
platforms to a detail that it can cope with all the different ways of linking
against them.

E.g. libxml can be built with icu, lzma, zlib support. If libxml is built
statically, postgres needs to link to all those libraries as well.  How can we
know which of those dependencies are enabled?


Even if we can make that somehow work, it's not reasonable for postgres
developers adding a dependency to have to figure out how to probe all of this,
when literally no other platform works that way anymore.


If you look around at recipes for building postgres on windows, they all have
to patch src/tools/msvc (see links at the bottom), because the builtin paths
and flags just don't work outside of a single way of acquiring the dependency.



The fact that this thread started only now is actually a good example for how
broken the approach to internalize all knowledge about building against
libraries into postgres is. This could all have been figured out 1+ years ago
- but wasn't.

Unless you want to require postgres devs to get constantly in the muck on
windows, we'll never get that right until just before the release.



I don't particularly care how we abstract away the low level linking details
on windows. We can use pkgconf, we can use cmake, we can invent our own thing.
But it has to be something other than hardcoding windows library paths and
compiler flags into our buildsystem.


And yes, that might make it a bit harder for a packager on windows, but
windows is already a *massive* drag on PG developers, it has to be somewhat
manageable.

I do think we can make the effort of windows dependency management a lot more
reasonable than it is now though, by providing a recipe for acquiring the
dependency in some form. It's a lot easier to for packagers and developers to
customize ontop of something like that.


Frankly, the fact that there's pretty much no automated testing of the various
dependencies that's accessible to non-windows devs is not a sustainable
situation.


> > Btw, I've been working with Bilal to add a few of the dependencies to the
> > CI
> > images so we can test those automatically. Using vcpkg. We got that nearly
> > working, but he's on vacation this week...  That does ensure both cmake and
> > .pc files are generated, fwiw.
> >
> > Currently builds gettext, icu, libxml2, libxslt, lz4, openssl, pkgconf,
> > python3, tcl, zlib, zstd.
>
>
> That appears to be using Mingw/Msys, which is quite different from a VC++
> build, in part because it's a full environment with its own package manager
> and packages that people have put a lot of effort into making work as they
> do on unix.

Err, that was a copy-paste mistake on my end and doesn't even use the vcpkg
generated stuff.

Here's an example build with most dependencies enabled (see below for more
details):

https://cirrus-ci.com/task/6497321108635648?logs=configure#L323


I started hacking a bit further on testing all dependencies, which led me down
a few rabbitholes:


- kerberos: When built linking against a debug runtime, it spews *ginormous*
  amounts of information onto stderr. Unfortunately its buildsystem doesn't
  seperate out debugging output and linking against a debug runtime. Argh.

  The tests fail even with a non-debug runtime though, due to debugging output
  in some cases, not sure why:
  https://cirrus-ci.com/task/5872684519653376?logs=check_world#L502

  Separately, the kerberos tests don't seem to be prepared to work on windows
  :(.

  So I disabled using it in CI for now.


- Linking the backend dynamically against lz4, icu, ssl, xml, xslt, zstd, zlib
  slows down the tests noticeably (~20%).  So I ended up building those
  statically.

  I ran into some issue with using a static libintl. I made it work, but for
  now reverted to a dynamic one.


- Enabling nls slows down the tests by about 15%, somewhat painful. This is
  when statically linking, it's a bit worse when linked dynamically :(.


- readline: Instead of the old issue with a compiler error, now we get a
  compiler crash:
  https://developercommunity.visualstudio.com/t/tab-completec4023:-fatal-error-C1001:/10685868

  The issue is fairly trivial to work around, we just need to break the the
  if/else chain into two. Probably deserves a bigger refactoring, but that's
  for another day.


I haven't yet looked into a) uuid b) tcl.  I think those are the only other
missing dependencies.


> > Many of them do include at least cmake files on windows if you build them
> > though?

> The only one that does is libxml2 as far as I can see. And that doesn't
> seem to work even if I use --cmake-prefix-path= as you suggested:

Ugh, that's because they used a different name for their cmake dependency than
for pkg-config. We can add the alternative spelling to meson.build.



>
> > > And that's why we really need to be able to locate headers and libraries
> > > easily by passing paths to meson, as we can't rely on pkgconfig, cmake,
> > or
> > > things being in some standard directory on Windows.
> >
> > Except that that often causes hard to diagnose breakages, because that
> > doesn't allow including the necessary compiler/linker flags [2].  It's a
> > bad model, we shouldn't perpetuate it.  If we want to forever make windows
> > a complicated annoying stepchild, that's the way to go.
>
> That is a good point, though I suspect it wouldn't solve your second
> example of the Kerberos libraries, as you'll get both 32 and 64 bit libs if
> you follow their standard process for building on Windows so you still need
> to have code to pick the right ones.

vcpkg for one does provide .pc files for kerberos.


> > FWIW, at least libzstd, libxml [3], lz4, zlib can generate cmake dependency
> > files on windows in their upstream code.
> >
>
> In the case of zstd, it does not if you build with VC++, the Makefile, or
> Meson, at least in my testing.

When building with meson it does generate a .pc file, which does work with
PG as-is.


> It looks like it would if you built it with cmake, but I couldn't get that
> to work in 10 minutes or so of messing around.


> And that's a perfect example of what I'm bleating about - there are often
> many ways of building things on Windows and there are definitely many ways
> of getting things on Windows, and they're not all equal.

Right - but that's precisely which is why it's unreasable for postgres to know
all the ins and outs of the different file locations and compiler flags for
all those sources. Hence needing to abstract that.


> We've either got to be extremely prescriptive in our docs, telling people
> precisely what they need to download for each dependency, or how to build it
> themselves in the way that will work with PostgreSQL, or the build system
> needs to be flexible enough to handle different dependency variations, as
> the old VC++ build system was.

I'm confused - the old build system wasn't flexible around this stuff *at
all*. Everyone had to patch it to get dependencies to work, unless you chose
exactly the right source to download from - which was often not documented or
outdated.

For example:
- https://github.com/microsoft/vcpkg/blob/master/ports/libpq/windows/msbuild.patch
- https://github.com/conan-io/conan-center-index/blob/1b24f7c74994ec6573e322b7ae4111c10f620ffa/recipes/libpq/all/conanfile.py#L116-L160
- https://github.com/conda-forge/postgresql-feedstock/tree/main/recipe/patches

Greetings,

Andres Freund

  7. Re: Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-06-21T11:20:49Z

    Hi

On Thu, 20 Jun 2024 at 21:58, Andres Freund <andres@anarazel.de> wrote:

>
> > I don't think requiring or expecting vcpkg or conan is reasonable at all,
> > for a number of reasons:
> >
> > - Neither supports all the dependencies at present.
> > - There are real supply chain verification concerns for vendors.
> > - That would be a huge change from what we've required in the last 19
> > years, with no deprecation notices or warnings for packagers etc.
>
> I don't think we should hard-require one specifically. I do think it'd be
> good
> if we provided an easy recipe for dependencies to be installed though.
>

That is precisely what https://github.com/dpage/winpgbuild/ was intended
for - and it works well for PG <= 16.


> I think such flexibility acually means it becomes *more* important to
> abstract
> away some of the concrete ways of using the various dependencies. It
> doesn't
> make sense for postgres to understand the internals of each dependency on
> all
> platforms to a detail that it can cope with all the different ways of
> linking
> against them.
>
> E.g. libxml can be built with icu, lzma, zlib support. If libxml is built
> statically, postgres needs to link to all those libraries as well.  How
> can we
> know which of those dependencies are enabled?
>

I don't think it's unreasonable to not support static linking, but I take
your point.


> Even if we can make that somehow work, it's not reasonable for postgres
> developers adding a dependency to have to figure out how to probe all of
> this,
> when literally no other platform works that way anymore.
>
> If you look around at recipes for building postgres on windows, they all
> have
> to patch src/tools/msvc (see links at the bottom), because the builtin
> paths
> and flags just don't work outside of a single way of acquiring the
> dependency.
>

I've been responsible for the Windows installers at EDB since we started
work on them, and prior to that built the original ones with Magnus. Since
v8.0, I've implemented multiple frameworks for building those packages, and
for building PostgreSQL as a dependency of other things (e.g. pgAdmin).
I've done that using builds of dependencies found at random places on the
internet, building them all myself, and a mixture.

I have never once had to patch the MSVC build system. The most I've ever
had to do is copy/rename a .lib file - zlib, which for some reason uses
different naming depending on how you build it. I vaguely recall that
OpenSSL had a similar issue in the distant past.

My point is that we seem to be heading from minor hacks to get things
working in some corner cases, towards requiring packagers and other people
building PostgreSQL on Windows having to do significant work to make the
dependencies look as we now expect. I suspect even more people will end up
patching the Meson build system as it might actually be easier to get
things to work.

The fact that this thread started only now is actually a good example for
> how
> broken the approach to internalize all knowledge about building against
> libraries into postgres is. This could all have been figured out 1+ years
> ago
> - but wasn't.
>
> Unless you want to require postgres devs to get constantly in the muck on
> windows, we'll never get that right until just before the release.
>

<rant>
Right now I'd be happy to just have the old MSVC build system back until we
can figure out a less complicated way to get to Meson (which I fully
support).

My assumption all along was that Meson would replace autoconf etc. before
anything happened with MSVC, precisely because that's the type of
environment all the Postgres devs work in primarily. Instead we seem to
have taken what I think is a flawed approach of entirely replacing the
build system on the platform none of the devs use, whilst leaving the new
system as an experimental option on the platforms it will have had orders
of magnitude more testing.

What makes it worse, is that I don't believe anyone was warned about such a
drastic change. Packagers were told about the git archive changes to the
tarball generation, but that's it (and we've said before, we can't expect
packagers to follow all the activity on -hackers).
</rant>


> I don't particularly care how we abstract away the low level linking
> details
> on windows. We can use pkgconf, we can use cmake, we can invent our own
> thing.
> But it has to be something other than hardcoding windows library paths and
> compiler flags into our buildsystem.
>
>
> And yes, that might make it a bit harder for a packager on windows, but
> windows is already a *massive* drag on PG developers, it has to be somewhat
> manageable.
>
> I do think we can make the effort of windows dependency management a lot
> more
> reasonable than it is now though, by providing a recipe for acquiring the
> dependency in some form. It's a lot easier to for packagers and developers
> to
> customize ontop of something like that.
>

Well as I noted, that is the point of my Github repo above. You can just go
download the binaries from the all_deps action - you can even download the
config.pl and buildenv.pl that will work with those dependencies (those
files are artefacts of the postgresql action).

We/I *could* add cmake/pc file generation to that tool, which would make
things work nicely with PostgreSQL 17 of course - however my original aim
for the project was to build all the dependencies in their officially
documented way, using MSVC (or UCRT64 if MSVC can't be used) for maximum
compatibility with the PG build, specifically eliminating or at least
minimising any custom build steps/hacks. As it turns out, I think the only
hack I really have is to avoid having to do an otherwise unnecessary 32bit
build of krb5.

Err, that was a copy-paste mistake on my end and doesn't even use the vcpkg
> generated stuff.
>
> Here's an example build with most dependencies enabled (see below for more
> details):
>
> https://cirrus-ci.com/task/6497321108635648?logs=configure#L323


OK.


> I started hacking a bit further on testing all dependencies, which led me
> down
> a few rabbitholes:
>
>
> - kerberos: When built linking against a debug runtime, it spews
> *ginormous*
>   amounts of information onto stderr. Unfortunately its buildsystem doesn't
>   seperate out debugging output and linking against a debug runtime. Argh.
>
>   The tests fail even with a non-debug runtime though, due to debugging
> output
>   in some cases, not sure why:
>   https://cirrus-ci.com/task/5872684519653376?logs=check_world#L502
>
>   Separately, the kerberos tests don't seem to be prepared to work on
> windows
>   :(.
>
>   So I disabled using it in CI for now.
>

Urgh, makes sense.


>
>
> - Linking the backend dynamically against lz4, icu, ssl, xml, xslt, zstd,
> zlib
>   slows down the tests noticeably (~20%).  So I ended up building those
>   statically.
>

Curious. I wonder if that translates into a general 20% performance hit.
Presumably it would for anything that looks similar to whatever test/tests
are affected.


>
>   I ran into some issue with using a static libintl. I made it work, but
> for
>   now reverted to a dynamic one.
>
>
> - Enabling nls slows down the tests by about 15%, somewhat painful. This is
>   when statically linking, it's a bit worse when linked dynamically :(.
>

That one I can imagine being in psql, so maybe not a big issue for most
real world use cases.


>
>
> - readline: Instead of the old issue with a compiler error, now we get a
>   compiler crash:
>
> https://developercommunity.visualstudio.com/t/tab-completec4023:-fatal-error-C1001:/10685868
>
>   The issue is fairly trivial to work around, we just need to break the the
>   if/else chain into two. Probably deserves a bigger refactoring, but
> that's
>   for another day.


>
> I haven't yet looked into a) uuid b) tcl.  I think those are the only other
> missing dependencies.
>

We really need to replace ossp-uuid on Windows anyway. It's basically
abandoned these days. I haven't looked to see if the alternatives work on
Windows now.


>
>
> > > Many of them do include at least cmake files on windows if you build
> them
> > > though?
>
> > The only one that does is libxml2 as far as I can see. And that doesn't
> > seem to work even if I use --cmake-prefix-path= as you suggested:
>
> Ugh, that's because they used a different name for their cmake dependency
> than
> for pkg-config. We can add the alternative spelling to meson.build.
>
>
>
> >
> > > > And that's why we really need to be able to locate headers and
> libraries
> > > > easily by passing paths to meson, as we can't rely on pkgconfig,
> cmake,
> > > or
> > > > things being in some standard directory on Windows.
> > >
> > > Except that that often causes hard to diagnose breakages, because that
> > > doesn't allow including the necessary compiler/linker flags [2].  It's
> a
> > > bad model, we shouldn't perpetuate it.  If we want to forever make
> windows
> > > a complicated annoying stepchild, that's the way to go.
> >
> > That is a good point, though I suspect it wouldn't solve your second
> > example of the Kerberos libraries, as you'll get both 32 and 64 bit libs
> if
> > you follow their standard process for building on Windows so you still
> need
> > to have code to pick the right ones.
>
> vcpkg for one does provide .pc files for kerberos.
>

Yes - that's in the vcpkg repo. I suspect they're adding pc and cmake files
for a lot of things.


> > We've either got to be extremely prescriptive in our docs, telling people
> > precisely what they need to download for each dependency, or how to
> build it
> > themselves in the way that will work with PostgreSQL, or the build system
> > needs to be flexible enough to handle different dependency variations, as
> > the old VC++ build system was.
>
> I'm confused - the old build system wasn't flexible around this stuff *at
> all*. Everyone had to patch it to get dependencies to work, unless you
> chose
> exactly the right source to download from - which was often not documented
> or
> outdated.
>

As I noted above - as the "owner" of the official packages, I never did
despite using a variety of upstream sources.


>
> For example:
> -
> https://github.com/microsoft/vcpkg/blob/master/ports/libpq/windows/msbuild.patch


That one looks almost entirely related to making PostgreSQL itself fit into
vcpkg's view of the world. It's changing the installation footprint, and
pulling some paths from their own variables. If they're changing our
installation footprint, it's likely they're doing the same for other
packages.


>
> -
> https://github.com/conan-io/conan-center-index/blob/1b24f7c74994ec6573e322b7ae4111c10f620ffa/recipes/libpq/all/conanfile.py#L116-L160


Same for that one. It's making many of those changes for non-Windows
platforms as well.


>
> -
> https://github.com/conda-forge/postgresql-feedstock/tree/main/recipe/patches


That one is interesting. It fixes the same zlib and OpenSSL issues I
mentioned being the one fix I did myself, albeit by renaming libraries
originally, and later by actually following the upstream build instructions
correctly.

It also makes essentially the same fix for krb5 that I hacked into my
Github Action, but similarly that isn't actually needed at all if you
follow the documented krb5 build process, which produces 32 and 64 bit
binaries.

Additionally, it also fixes a GSSAPI related bug which I reported a week or
two back here and for which there is a patch waiting to be committed, and
replaces some setenv calls with _putenv_.

There are a couple more patches in there, but they're Linux related from a
quick glance.


In short, I don't really see anything in those examples that are general
issues (aside from the bugs of course).


-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

  8. Re: Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-06-21T14:36:56Z

    Hi

On Fri, 21 Jun 2024 at 12:20, Dave Page <dpage@pgadmin.org> wrote:

>
> We/I *could* add cmake/pc file generation to that tool, which would make
> things work nicely with PostgreSQL 17 of course.
>

For giggles, I took a crack at doing that, manually creating .pc files for
everything I've been working with so far. It seems to work as expected,
except that unlike everything else libintl is detected entirely based on
whether the header and library can be found. I had to pass extra lib and
include dirs:

meson setup --wipe --pkg-config-path=C:\build64\lib\pkgconfig
-Dextra_include_dirs=C:\build64\include -Dextra_lib_dirs=C:\build64\lib
-Duuid=ossp build-auto

I'm assuming that's an oversight, given your previous comments?

-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

  9. Re: Meson far from ready on Windows

    Robert Haas <robertmhaas@gmail.com> — 2024-06-21T15:15:34Z

    On Fri, Jun 21, 2024 at 7:21 AM Dave Page <dpage@pgadmin.org> wrote:
> My assumption all along was that Meson would replace autoconf etc. before anything happened with MSVC, precisely because that's the type of environment all the Postgres devs work in primarily. Instead we seem to have taken what I think is a flawed approach of entirely replacing the build system on the platform none of the devs use, whilst leaving the new system as an experimental option on the platforms it will have had orders of magnitude more testing.
>
> What makes it worse, is that I don't believe anyone was warned about such a drastic change. Packagers were told about the git archive changes to the tarball generation, but that's it (and we've said before, we can't expect packagers to follow all the activity on -hackers).

I agree that we should have given a heads up to pgsql-packagers. The
fact that that wasn't done is a mistake, and inconsiderate. At the
same time, I don't quite know who should have done that exactly when.
Note that, while I believe Andres is on pgsql-packagers, many
committers are not, and we have no written guidelines anywhere for
what kinds of changes require notifying pgsql-packagers.

Previous threads on this issue:

https://postgr.es/m/ZQzp_VMJcerM1Cs_@paquier.xyz
http://postgr.es/m/20230408191007.7lysd42euafwl74f@awork3.anarazel.de

Note that in the second of these threads, which contemplated removing
MSVC for v16, I actually pointed out that if we went that way, we
needed to notify pgsql-packagers ASAP. But, since we didn't do that,
no email was ever sent to pgsql-packagers about this, or at least not
that I can find. Still, MSVC support was removed more than six months
ago, so even if somebody didn't see any of the pgsql-hackers
discussion about this, there's been a fair amount of time (and a beta)
for someone to notice that their build process isn't working any more.
It seems a bit weird to me to start complaining about this now.

As a practical matter, I don't think MSVC is coming back. The
buildfarm was already changed over to use meson, and it would be
pretty disruptive to try to re-add buildfarm coverage for a
resurrected MSVC on the eve of beta2. I think we should focus on
improving whatever isn't quite right in meson -- plenty of other
people have also complained about various things there, me included --
rather than trying to undo over a year's worth of work by lots of
people to get things on Windows switched over to MSVC.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  10. Re: Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-06-21T15:46:06Z

    On Fri, 21 Jun 2024 at 16:15, Robert Haas <robertmhaas@gmail.com> wrote:

> On Fri, Jun 21, 2024 at 7:21 AM Dave Page <dpage@pgadmin.org> wrote:
> > My assumption all along was that Meson would replace autoconf etc.
> before anything happened with MSVC, precisely because that's the type of
> environment all the Postgres devs work in primarily. Instead we seem to
> have taken what I think is a flawed approach of entirely replacing the
> build system on the platform none of the devs use, whilst leaving the new
> system as an experimental option on the platforms it will have had orders
> of magnitude more testing.
> >
> > What makes it worse, is that I don't believe anyone was warned about
> such a drastic change. Packagers were told about the git archive changes to
> the tarball generation, but that's it (and we've said before, we can't
> expect packagers to follow all the activity on -hackers).
>
> I agree that we should have given a heads up to pgsql-packagers. The
> fact that that wasn't done is a mistake, and inconsiderate. At the
> same time, I don't quite know who should have done that exactly when.
> Note that, while I believe Andres is on pgsql-packagers, many
> committers are not, and we have no written guidelines anywhere for
> what kinds of changes require notifying pgsql-packagers.
>
> Previous threads on this issue:
>
> https://postgr.es/m/ZQzp_VMJcerM1Cs_@paquier.xyz
> http://postgr.es/m/20230408191007.7lysd42euafwl74f@awork3.anarazel.de
>
> Note that in the second of these threads, which contemplated removing
> MSVC for v16, I actually pointed out that if we went that way, we
> needed to notify pgsql-packagers ASAP. But, since we didn't do that,
> no email was ever sent to pgsql-packagers about this, or at least not
> that I can find.


That's what I was saying should have been done. I don't think there was a
requirement on Andres to tell them that they could use Meson instead.


> Still, MSVC support was removed more than six months
> ago, so even if somebody didn't see any of the pgsql-hackers
> discussion about this, there's been a fair amount of time (and a beta)
> for someone to notice that their build process isn't working any more.
> It seems a bit weird to me to start complaining about this now.
>

People noticed when they started prepping for beta1. Then there was a mad
rush to get things working under Meson in any way possible.


> As a practical matter, I don't think MSVC is coming back. The
> buildfarm was already changed over to use meson, and it would be
> pretty disruptive to try to re-add buildfarm coverage for a
> resurrected MSVC on the eve of beta2. I think we should focus on
> improving whatever isn't quite right in meson -- plenty of other
> people have also complained about various things there, me included --
> rather than trying to undo over a year's worth of work by lots of
> people to get things on Windows switched over to MSVC.
>

The buildfarm hasn't switched over - it had support added for Meson. If it
had been switched, then the older back branches would have gone red.

Anyway, that's immaterial - I know the old code isn't coming back now. My
motivation for this thread is to get Meson to a usable state on Windows,
that doesn't require hacking stuff around for the casual builder moving
forwards - and at present, it requires *significantly* more hacking around
than it has in many years.

The design goals Andres spoke about would clearly be a technical
improvement to PostgreSQL, however as we're finding, they rely on the
upstream dependencies being built with pkgconfig or cmake files which
either doesn't happen at present, or only happens if you happen to build in
a certain way, or download from some source that has added them. I'm not
sure how to fix that without re-introducing the old hacks in the build
system, or extending my side project to add .pc files to all the
dependencies it builds. I will almost certainly do that, as it'll give
folks a single place where they can download everything they need, and
provide a reference on how everything can be built if they want to do it
themselves, but on the other hand, it's far from an ideal solution and I'd
much prefer if I didn't need to do that at all.

-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

  11. Re: Meson far from ready on Windows

    Tristan Partin <tristan@partin.io> — 2024-06-21T17:16:06Z

    On Fri Jun 21, 2024 at 10:46 AM CDT, Dave Page wrote:
> On Fri, 21 Jun 2024 at 16:15, Robert Haas <robertmhaas@gmail.com> wrote:
> > As a practical matter, I don't think MSVC is coming back. The
> > buildfarm was already changed over to use meson, and it would be
> > pretty disruptive to try to re-add buildfarm coverage for a
> > resurrected MSVC on the eve of beta2. I think we should focus on
> > improving whatever isn't quite right in meson -- plenty of other
> > people have also complained about various things there, me included --
> > rather than trying to undo over a year's worth of work by lots of
> > people to get things on Windows switched over to MSVC.
> >
>
> The buildfarm hasn't switched over - it had support added for Meson. If it
> had been switched, then the older back branches would have gone red.
>
> Anyway, that's immaterial - I know the old code isn't coming back now. My
> motivation for this thread is to get Meson to a usable state on Windows,
> that doesn't require hacking stuff around for the casual builder moving
> forwards - and at present, it requires *significantly* more hacking around
> than it has in many years.
>
> The design goals Andres spoke about would clearly be a technical
> improvement to PostgreSQL, however as we're finding, they rely on the
> upstream dependencies being built with pkgconfig or cmake files which
> either doesn't happen at present, or only happens if you happen to build in
> a certain way, or download from some source that has added them. I'm not
> sure how to fix that without re-introducing the old hacks in the build
> system, or extending my side project to add .pc files to all the
> dependencies it builds. I will almost certainly do that, as it'll give
> folks a single place where they can download everything they need, and
> provide a reference on how everything can be built if they want to do it
> themselves, but on the other hand, it's far from an ideal solution and I'd
> much prefer if I didn't need to do that at all.

Hey Dave,

I'm a maintainer for Meson, and am happy to help you in any way that 
I reasonably can.

Let's start with the state of Windows support in Meson. If I were to 
rank Meson support for platforms, I would do something like this:

- Linux
- BSDs
- Solaris/IllumOS
- ...
- Apple
- Windows

As you can see Windows is the bottom of the totem pole. We don't have 
Windows people coming along to contribute very often for whatever 
reason. Thus admittedly, Windows support can be very lackluster at 
times.

Meson is not a project which sees a lot of funding. (Do any build 
tools?) The projects that support Meson in any way are Mesa and 
GStreamer, which don't have a lot of incentive to do anything with 
Windows, generally.

I'm not even sure any of the regular contributors to Meson have 
Windows development machines. I surely don't have access to a Windows 
machine.

All that being said, I would like to help you solve your Windows 
dependencies issue, or at least mitigate them. I think it is probably 
best to just look at one dependency at a time. Here is how lz4 is picked 
up in the Postgres Meson build:

> lz4opt = get_option('lz4')
> if not lz4opt.disabled()
>   lz4 = dependency('liblz4', required: lz4opt)
> 
>   if lz4.found()
>     cdata.set('USE_LZ4', 1)
>     cdata.set('HAVE_LIBLZ4', 1)
>   endif
> 
> else
>   lz4 = not_found_dep
> endif

As you are well aware, dependency() looks largely at pkgconfig and cmake 
to find the dependencies. In your case, that is obviously not working. 

I think there are two ways to solve your problem. A first solution would 
look like this:

> lz4opt = get_option('lz4')
> if not lz4opt.disabled()
>   lz4 = dependency('liblz4', required: false)
>   if not lz4.found()
>     lz4 = cc.find_library('lz4', required: lz4opt, dirs: extra_lib_dirs)
>   endif
> 
>   if lz4.found()
>     cdata.set('USE_LZ4', 1)
>     cdata.set('HAVE_LIBLZ4', 1)
>   end
> else
>   lz4 = not_found_dep
> endif

Another solution that could work alongside the previous suggestion is to 
use Meson subprojects[0] for managing Postgres dependencies. I don't 
know if we would want this in the Postgres repo or a patch that 
downstream packagers would need to apply, but essentially, add the wrap 
file:

> [wrap-file]
> directory = lz4-1.9.4
> source_url = https://github.com/lz4/lz4/archive/v1.9.4.tar.gz
> source_filename = lz4-1.9.4.tgz
> source_hash = 0b0e3aa07c8c063ddf40b082bdf7e37a1562bda40a0ff5272957f3e987e0e54b
> patch_filename = lz4_1.9.4-2_patch.zip
> patch_url = https://wrapdb.mesonbuild.com/v2/lz4_1.9.4-2/get_patch
> patch_hash = 4f33456cce986167d23faf5d28a128e773746c10789950475d2155a7914630fb
> wrapdb_version = 1.9.4-2
> 
> [provide]
> liblz4 = liblz4_dep

into subprojects/lz4.wrap, and Meson should be able to automagically 
pick up the dependency. Do this for all the projects that Postgres 
depends on, and you'll have an entire build managed by Meson. Note that 
Meson subprojects don't have to use Meson themselves. They can also use 
CMake[1] or Autotools[2], but your results may vary.

Happy to hear your thoughts. I think if our goal is to enable more 
people to work on Postgres, we should probably add subproject wraps to 
the source tree, but we also need to be forgiving like in the Meson DSL 
snippet above.

Let me know your thoughts!

[0]: https://mesonbuild.com/Wrap-dependency-system-manual.html
[1]: https://github.com/hse-project/hse/blob/6d5207f88044a3bd9b3539260074395317e276d5/meson.build#L239-L275
[2]: https://github.com/hse-project/hse/blob/6d5207f88044a3bd9b3539260074395317e276d5/subprojects/packagefiles/libbsd/meson.build

-- 
Tristan Partin
https://tristan.partin.io

  12. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-06-22T16:32:25Z

    Hi,

On 2024-06-21 12:20:49 +0100, Dave Page wrote:
> On Thu, 20 Jun 2024 at 21:58, Andres Freund <andres@anarazel.de> wrote:
> That is precisely what https://github.com/dpage/winpgbuild/ was intended
> for - and it works well for PG <= 16.

If we develop it into that - I'd be happy. I mostly want to be able to do
automated tests on windows with all reasonable dependencies. And occasionally
do some interactive investigation, without a lot of setup time.

One small advantage of something outside of PG is that it's easy to add
additional dependencies when developing additional features. Except of course
all the windows packaging systems seem ... suboptimal.


> I don't think it's unreasonable to not support static linking, but I take
> your point.

Separately from this thread: ISTM that on windows it'd be quite beneficial to
link a few things statically, given how annoying dealing with dlls can be?
There's also the perf issue addressed further down.


> My assumption all along was that Meson would replace autoconf etc. before
> anything happened with MSVC, precisely because that's the type of
> environment all the Postgres devs work in primarily. Instead we seem to
> have taken what I think is a flawed approach of entirely replacing the
> build system on the platform none of the devs use, whilst leaving the new
> system as an experimental option on the platforms it will have had orders
> of magnitude more testing.

The old system was a major bottleneck. For one, there was no way to run all
tests. And even the tests that one could run, would run serially, leading to
exceedingly long tests times. While that could partially be addressed by
having both buildsystems in parallel, the old one would frequently break in a
way that one couldn't reproduce on other systems. And resource wise it wasn't
feasible to test both old and new system for cfbot/CI.


> What makes it worse, is that I don't believe anyone was warned about such a
> drastic change. Packagers were told about the git archive changes to the
> tarball generation, but that's it (and we've said before, we can't expect
> packagers to follow all the activity on -hackers).

I'm sure we could have dealt better with it. There certainly was some lack of
of cohesion because I wasn't able to do drive the src/tools/msvc removal and
Michael took up the slack.

But I also don't think it's really fair to say that there was no heads
up. Several people at EDB participated in the removal and buildfarm
maintainers at EDB were repeatedly pinged, to move their buildfarm animals
over.

And of course the meson stuff came out a year earlier and it wouldn't have
been exactly unreasonable


> Well as I noted, that is the point of my Github repo above. You can just go
> download the binaries from the all_deps action - you can even download the
> config.pl and buildenv.pl that will work with those dependencies (those
> files are artefacts of the postgresql action).

For the purpose of CI we'd really need debug builds of most of the libraries -
there are compat issues when mixing debug/non-debug runtimes (we've hit them
occasionally) and not using the debug runtime hides a lot of issues. Of course
also not optimal for CI / BF usage.



> > - Linking the backend dynamically against lz4, icu, ssl, xml, xslt, zstd,
> > zlib
> >   slows down the tests noticeably (~20%).  So I ended up building those
> >   statically.

> Curious. I wonder if that translates into a general 20% performance hit.
> Presumably it would for anything that looks similar to whatever test/tests
> are affected.

FWIW, dynamic linking has a noticeable overhead on other platforms too. A
non-dependencies-enabled postgres can do about 2x the connections-per-second
than a fully kitted out postgres can (basically due to more memory mapping
metadata being copied).  But on windows the overhead is larger because so much
more happens for every new connections, including loading all dlls from
scratch.

I suspect linking a few libraries statically would be quite worth it on
windows. On other platforms it'd be quite inadvisable to statically link
libraries, due to security updates, but for stuff like the EDB windows
installer dynamic linking doesn't really help with that afaict?



> >   I ran into some issue with using a static libintl. I made it work, but
> > for
> >   now reverted to a dynamic one.
> >
> >
> > - Enabling nls slows down the tests by about 15%, somewhat painful. This is
> >   when statically linking, it's a bit worse when linked dynamically :(.
> >
> 
> That one I can imagine being in psql, so maybe not a big issue for most
> real world use cases.

I think it's both psql and backend.  I think partially it's just due the
additional libraries being linked in everywhere (intl and iconv) and partially
it's the additinal indirection that happens in a bunch more places. We have a
bunch of places where we do gettext lookups but never use the result unless
you use DEBUG3 or such, and that not free. It also triggers additional
filesystem lookups (for the translations) and that's not cheap on windows
either.

> >
> > I haven't yet looked into a) uuid b) tcl.  I think those are the only other
> > missing dependencies.
> >
> 
> We really need to replace ossp-uuid on Windows anyway. It's basically
> abandoned these days. I haven't looked to see if the alternatives work on
> Windows now.

Yea, once we have *something* for ossp-uuid, I think we should remove support
for ossp-uuid. We don't do anyone favors by inducing them to install it.


Greetings,

Andres Freund

  13. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-06-22T16:35:00Z

    Hi,

On 2024-06-21 15:36:56 +0100, Dave Page wrote:
> For giggles, I took a crack at doing that, manually creating .pc files for
> everything I've been working with so far.

Cool!


> It seems to work as expected, except that unlike everything else libintl is
> detected entirely based on whether the header and library can be found. I
> had to pass extra lib and include dirs:

Yea, right now libintl isn't using dependency detection because I didn't see
any platform where it's distributed with a .pc for or such. It'd be just a
line or two to make it use one...

Greetings,

Andres Freund

  14. Re: Meson far from ready on Windows

    Wolfgang Walther <walther@technowledgy.de> — 2024-06-22T17:32:01Z

    Andres Freund:
> FWIW, dynamic linking has a noticeable overhead on other platforms too. A
> non-dependencies-enabled postgres can do about 2x the connections-per-second
> than a fully kitted out postgres can (basically due to more memory mapping
> metadata being copied).  But on windows the overhead is larger because so much
> more happens for every new connections, including loading all dlls from
> scratch.
> 
> I suspect linking a few libraries statically would be quite worth it on
> windows. On other platforms it'd be quite inadvisable to statically link
> libraries, due to security updates, [...]

That's not necessarily true. The nix package manager and thus NixOS 
track all dependencies for a piece of software. If any of the 
dependencies are updated, all dependents are rebuilt, too. So the 
security concern doesn't apply here. There is a "static overlay", which 
builds everything linked fully statically. Unfortunately, PostgreSQL 
doesn't build in that, so far.

Lately, I have been looking into building at least libpq in that static 
overlay, via Meson. There are two related config options:
-Ddefault_library=shared|static|both
-Dprefer_static

The first controls which libraries (libpq, ...) to build ourselves. The 
second controls linking, IIUC also against external dependencies.

Maybe it would be a first step to support -Dprefer_static?

Then this could be set on Windows.

Best,

Wolfgang

  15. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-06-22T21:17:43Z

    Hi, 

On June 22, 2024 7:32:01 PM GMT+02:00, walther@technowledgy.de wrote:
>Andres Freund:
>> FWIW, dynamic linking has a noticeable overhead on other platforms too. A
>> non-dependencies-enabled postgres can do about 2x the connections-per-second
>> than a fully kitted out postgres can (basically due to more memory mapping
>> metadata being copied).  But on windows the overhead is larger because so much
>> more happens for every new connections, including loading all dlls from
>> scratch.
>> 
>> I suspect linking a few libraries statically would be quite worth it on
>> windows. On other platforms it'd be quite inadvisable to statically link
>> libraries, due to security updates, [...]
>That's not necessarily true. The nix package manager and thus NixOS track all dependencies for a piece of software. If any of the dependencies are updated, all dependents are rebuilt, too. So the security concern doesn't apply here. There is a "static overlay", which builds everything linked fully statically. 

Right. There's definitely some scenario where it's ok, I was simplifying a bit.

> Unfortunately, PostgreSQL doesn't build in that, so far.

I've built mostly statically linked pg without much of a problem, what trouble did you encounter? Think there were some issues with linking Kerberos and openldap statically, but not on postgres' side.

Building the postgres backend without support for dynamic linking doesn't make sense though. Extensions are just stop ingrained part of pg.


>Lately, I have been looking into building at least libpq in that static overlay, via Meson. There are two related config options:
>-Ddefault_library=shared|static|both
>-Dprefer_static
>
>The first controls which libraries (libpq, ...) to build ourselves. The second controls linking, IIUC also against external dependencies.

Pg by default builds a static libpq on nearly all platforms (not aix I think and maybe not Windows when building with autoconf, not sure about the old msvc system) today?


>Maybe it would be a first step to support -Dprefer_static?

That should work for nearly all dependencies today. Except for libintl, I think.  I found that there are a lot of buglets in static link dependencies of various libraries though. 


Andres
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

  16. Re: Meson far from ready on Windows

    Andrew Dunstan <andrew@dunslane.net> — 2024-06-23T10:18:44Z

    On 2024-06-21 Fr 11:15 AM, Robert Haas wrote:
> As a practical matter, I don't think MSVC is coming back. The
> buildfarm was already changed over to use meson, and it would be
> pretty disruptive to try to re-add buildfarm coverage for a
> resurrected MSVC on the eve of beta2. I think we should focus on
> improving whatever isn't quite right in meson -- plenty of other
> people have also complained about various things there, me included --
> rather than trying to undo over a year's worth of work by lots of
> people to get things on Windows switched over to MSVC.
>

As a practical matter, whether the buildfarm client uses meson or not is 
a matter of one line in the client's config file. Support for the old 
system is still there, of course, as it's required on older branches. So 
the impact would be pretty minimal if we did decide to re-enable the old 
build system. There are only two MSVC animals building master right now: 
drongo (run by me) and hammerkop (run by our friends at SRA OSS).

I am not necessarily advocating it, just setting the record straight 
about how  easy it would be to switch the buildfarm.


cheers


andrew




--
Andrew Dunstan
EDB:https://www.enterprisedb.com

  17. Re: Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-06-24T08:44:57Z

    Hi Tristan,

On Fri, 21 Jun 2024 at 18:16, Tristan Partin <tristan@partin.io> wrote:

>
> Hey Dave,
>
> I'm a maintainer for Meson, and am happy to help you in any way that
> I reasonably can.
>

Thank you!

>
> Let's start with the state of Windows support in Meson. If I were to
> rank Meson support for platforms, I would do something like this:
>
> - Linux
> - BSDs
> - Solaris/IllumOS
> - ...
> - Apple
> - Windows
>
> As you can see Windows is the bottom of the totem pole. We don't have
> Windows people coming along to contribute very often for whatever
> reason. Thus admittedly, Windows support can be very lackluster at
> times.
>
> Meson is not a project which sees a lot of funding. (Do any build
> tools?) The projects that support Meson in any way are Mesa and
> GStreamer, which don't have a lot of incentive to do anything with
> Windows, generally.
>
> I'm not even sure any of the regular contributors to Meson have
> Windows development machines. I surely don't have access to a Windows
> machine.
>

To be very clear, my comments - in particular the subject line of this
thread - are not referring to Meson itself, rather our use of it on
Windows. I've been quite impressed with Meson in general, and am coming to
like it a lot.


>
> All that being said, I would like to help you solve your Windows
> dependencies issue, or at least mitigate them. I think it is probably
> best to just look at one dependency at a time. Here is how lz4 is picked
> up in the Postgres Meson build:
>
> > lz4opt = get_option('lz4')
> > if not lz4opt.disabled()
> >   lz4 = dependency('liblz4', required: lz4opt)
> >
> >   if lz4.found()
> >     cdata.set('USE_LZ4', 1)
> >     cdata.set('HAVE_LIBLZ4', 1)
> >   endif
> >
> > else
> >   lz4 = not_found_dep
> > endif
>
> As you are well aware, dependency() looks largely at pkgconfig and cmake
> to find the dependencies. In your case, that is obviously not working.
>
> I think there are two ways to solve your problem. A first solution would
> look like this:
>
> > lz4opt = get_option('lz4')
> > if not lz4opt.disabled()
> >   lz4 = dependency('liblz4', required: false)
> >   if not lz4.found()
> >     lz4 = cc.find_library('lz4', required: lz4opt, dirs: extra_lib_dirs)
> >   endif
> >
> >   if lz4.found()
> >     cdata.set('USE_LZ4', 1)
> >     cdata.set('HAVE_LIBLZ4', 1)
> >   end
> > else
> >   lz4 = not_found_dep
> > endif
>

Yes, that's the pattern I think we should generally be using:

- It supports the design goals, allowing for configurations we don't know
about to be communicated through pkgconfig or cmake files.
- It provides a fallback method to detect the dependencies as we do in the
old MSVC build system, which should work with most dependencies built with
their "standard" configuration on Windows.

To address Andres' concerns around mis-detection of dependencies, or other
oddities such as required compiler flags not being included, I would
suggest that a) that's happened very rarely, if ever, in the past, and b)
we can always spit out an obvious warning if we've not been able to use
cmake or pkgconfig for any particular dependencies.


>
> Another solution that could work alongside the previous suggestion is to
> use Meson subprojects[0] for managing Postgres dependencies. I don't
> know if we would want this in the Postgres repo or a patch that
> downstream packagers would need to apply, but essentially, add the wrap
> file:
>
> > [wrap-file]
> > directory = lz4-1.9.4
> > source_url = https://github.com/lz4/lz4/archive/v1.9.4.tar.gz
> > source_filename = lz4-1.9.4.tgz
> > source_hash =
> 0b0e3aa07c8c063ddf40b082bdf7e37a1562bda40a0ff5272957f3e987e0e54b
> > patch_filename = lz4_1.9.4-2_patch.zip
> > patch_url = https://wrapdb.mesonbuild.com/v2/lz4_1.9.4-2/get_patch
> > patch_hash =
> 4f33456cce986167d23faf5d28a128e773746c10789950475d2155a7914630fb
> > wrapdb_version = 1.9.4-2
> >
> > [provide]
> > liblz4 = liblz4_dep
>
> into subprojects/lz4.wrap, and Meson should be able to automagically
> pick up the dependency. Do this for all the projects that Postgres
> depends on, and you'll have an entire build managed by Meson. Note that
> Meson subprojects don't have to use Meson themselves. They can also use
> CMake[1] or Autotools[2], but your results may vary.
>

That's certainly interesting functionality. I'm not sure we'd want to try
to use it here, simply because building all of PostgreSQL's dependencies on
Windows requires multiple different toolchains and environments and is not
trivial to setup. That's largely why I started working on
https://github.com/dpage/winpgbuild.

Thanks!


>
> Happy to hear your thoughts. I think if our goal is to enable more
> people to work on Postgres, we should probably add subproject wraps to
> the source tree, but we also need to be forgiving like in the Meson DSL
> snippet above.
>
> Let me know your thoughts!
>
> [0]: https://mesonbuild.com/Wrap-dependency-system-manual.html
> [1]:
> https://github.com/hse-project/hse/blob/6d5207f88044a3bd9b3539260074395317e276d5/meson.build#L239-L275
> [2]:
> https://github.com/hse-project/hse/blob/6d5207f88044a3bd9b3539260074395317e276d5/subprojects/packagefiles/libbsd/meson.build
>
> --
> Tristan Partin
> https://tristan.partin.io
>


-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

  18. Re: Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-06-24T08:54:51Z

    Hi

On Sat, 22 Jun 2024 at 17:32, Andres Freund <andres@anarazel.de> wrote:

> > I don't think it's unreasonable to not support static linking, but I take
> > your point.
>
> Separately from this thread: ISTM that on windows it'd be quite beneficial
> to
> link a few things statically, given how annoying dealing with dlls can be?
> There's also the perf issue addressed further down.
>

Dealing with DLLs largely just boils down to copying them into the right
place when packaging. The perf issue is a much more compelling reason to
look at static linking imho.


>
>
> > My assumption all along was that Meson would replace autoconf etc. before
> > anything happened with MSVC, precisely because that's the type of
> > environment all the Postgres devs work in primarily. Instead we seem to
> > have taken what I think is a flawed approach of entirely replacing the
> > build system on the platform none of the devs use, whilst leaving the new
> > system as an experimental option on the platforms it will have had orders
> > of magnitude more testing.
>
> The old system was a major bottleneck. For one, there was no way to run all
> tests. And even the tests that one could run, would run serially, leading
> to
> exceedingly long tests times. While that could partially be addressed by
> having both buildsystems in parallel, the old one would frequently break
> in a
> way that one couldn't reproduce on other systems. And resource wise it
> wasn't
> feasible to test both old and new system for cfbot/CI.
>

Hmm, I've found that running the tests under Meson takes notably longer
than the old system - maybe 5 - 10x longer ("meson test" vs. "vcregress
check"). I haven't yet put any effort into figuring out a cause for that
yet.


> FWIW, dynamic linking has a noticeable overhead on other platforms too. A
> non-dependencies-enabled postgres can do about 2x the
> connections-per-second
> than a fully kitted out postgres can (basically due to more memory mapping
> metadata being copied).  But on windows the overhead is larger because so
> much
> more happens for every new connections, including loading all dlls from
> scratch.
>
> I suspect linking a few libraries statically would be quite worth it on
> windows. On other platforms it'd be quite inadvisable to statically link
> libraries, due to security updates, but for stuff like the EDB windows
> installer dynamic linking doesn't really help with that afaict?
>

Correct - we're shipping the dependencies ourselves, so we have to
rewrap/retest anyway.

-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

  19. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-06-24T09:12:02Z

    Hi,

On 2024-06-24 09:54:51 +0100, Dave Page wrote:
> > The old system was a major bottleneck. For one, there was no way to run all
> > tests. And even the tests that one could run, would run serially, leading
> > to
> > exceedingly long tests times. While that could partially be addressed by
> > having both buildsystems in parallel, the old one would frequently break
> > in a
> > way that one couldn't reproduce on other systems. And resource wise it
> > wasn't
> > feasible to test both old and new system for cfbot/CI.
> >
> 
> Hmm, I've found that running the tests under Meson takes notably longer
> than the old system - maybe 5 - 10x longer ("meson test" vs. "vcregress
> check"). I haven't yet put any effort into figuring out a cause for that
> yet.

That's because vcregress check only runs a small portion of the tests (just
the main pg_regress checks, no tap tests, no extension). Which is pretty much
my point.

To run a decent, but still far from complete, portion of the tests you needed to do
this:
https://github.com/postgres/postgres/blob/REL_15_STABLE/.cirrus.tasks.yml#L402-L440

If you want to run just the main regression tests with meson, you can:
  meson test --suite setup --suite regress

To see the list of all tests
  meson test --list

Greetings,

Andres Freund

  20. Re: Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-06-24T12:24:05Z

    On Sat, 22 Jun 2024 at 17:35, Andres Freund <andres@anarazel.de> wrote:

> Hi,
>
> On 2024-06-21 15:36:56 +0100, Dave Page wrote:
> > For giggles, I took a crack at doing that, manually creating .pc files
> for
> > everything I've been working with so far.
>
> Cool!
>
>
> > It seems to work as expected, except that unlike everything else libintl
> is
> > detected entirely based on whether the header and library can be found. I
> > had to pass extra lib and include dirs:
>
> Yea, right now libintl isn't using dependency detection because I didn't
> see
> any platform where it's distributed with a .pc for or such. It'd be just a
> line or two to make it use one...
>

I think it should, for consistency if nothing else - especially if we're
adding our own pc/cmake files to prebuilt dependencies.

-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

  21. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-06-25T10:41:45Z

    Hi,

On 2024-06-21 12:20:49 +0100, Dave Page wrote:
> > I'm confused - the old build system wasn't flexible around this stuff *at
> > all*. Everyone had to patch it to get dependencies to work, unless you
> > chose
> > exactly the right source to download from - which was often not documented
> > or
> > outdated.
> >
> 
> As I noted above - as the "owner" of the official packages, I never did
> despite using a variety of upstream sources.

For reference, with 16 and src/tools/msvc:
- upstream zstd build doesn't work, wrong filename (libzstd.dll.a instead of libzstd.lib)
- upstream lz4 build doesn't work, wrong filename (liblz4.dll.a instead of liblz4.lib)
- openssl, from https://slproweb.com/products/Win32OpenSSL.htm , as our
  docs suggest: doesn't work, wrong filenames (openssl.lib instead of
  lib*64.lib, works if you delete lib/VC/sslcrypto64MD.lib)
- iconv/intl: mismatching library names (lib*.dll.a lib*.lib)

- zlib at least at least from some of the sources (it's hard to tell, because
  everything available is so outdated), wrong filenames

Upstream ICU works.

I gave up at this point, so I don't know if libxml, xslt and uuid work without
patching the sources.

Greetings,

Andres Freund

  22. Re: Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-06-25T10:54:56Z

    On Tue, 25 Jun 2024 at 11:41, Andres Freund <andres@anarazel.de> wrote:

> Hi,
>
> On 2024-06-21 12:20:49 +0100, Dave Page wrote:
> > > I'm confused - the old build system wasn't flexible around this stuff
> *at
> > > all*. Everyone had to patch it to get dependencies to work, unless you
> > > chose
> > > exactly the right source to download from - which was often not
> documented
> > > or
> > > outdated.
> > >
> >
> > As I noted above - as the "owner" of the official packages, I never did
> > despite using a variety of upstream sources.
>
> For reference, with 16 and src/tools/msvc:
> - upstream zstd build doesn't work, wrong filename (libzstd.dll.a instead
> of libzstd.lib)
> - upstream lz4 build doesn't work, wrong filename (liblz4.dll.a instead of
> liblz4.lib)
> - openssl, from https://slproweb.com/products/Win32OpenSSL.htm , as our
>   docs suggest: doesn't work, wrong filenames (openssl.lib instead of
>   lib*64.lib, works if you delete lib/VC/sslcrypto64MD.lib)
> - iconv/intl: mismatching library names (lib*.dll.a lib*.lib)
>
> - zlib at least at least from some of the sources (it's hard to tell,
> because
>   everything available is so outdated), wrong filenames
>

https://github.com/dpage/winpgbuild proves that the hacks above are not
required *if* you build the dependencies in the recommended way for use
with MSVC++ (where documented), otherwise just native Windows.

If you, for example, build a dependency using Mingw/Msys, then you may get
different filenames than if you build the same thing using its VC++
solution or makefile. That's where most, if not all, of these issues come
from.

It's probably worth noting that "back in the day" when most of this stuff
was built, there was no UCRT32 compiler option, and it really was a
potential problem to mix VC++ and Mingw compiled binaries so there was a
heavy focus on making sure everything was designed around the MSVC++ builds
wherever they existed.

-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

  23. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-06-25T11:23:37Z

    Hi,

On 2024-06-24 09:44:57 +0100, Dave Page wrote:
> To address Andres' concerns around mis-detection of dependencies, or other
> oddities such as required compiler flags not being included, I would
> suggest that a) that's happened very rarely, if ever, in the past, and b)
> we can always spit out an obvious warning if we've not been able to use
> cmake or pkgconfig for any particular dependencies.

I personally spent quite a few days hunting down issues related to this. Not
because I wanted to, but because it was causing breakage and nobody else was
looking.  For several years postgres didn't build against a modern perl, for
example, see the stuff leading up to
  http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=ccc59a83cd97
but nobody seemed to care for a prolonged amount of time.

We have evidence of random build hackery all over the tree - often
entirely outdated, sometimes even *breaking* builds these days ([1]):

https://github.com/postgres/postgres/blob/master/src/interfaces/libpq/Makefile#L80-L88
(wrong library names for kerberos on 64bit systems, wrong ssl libnames for )

https://github.com/postgres/postgres/blob/master/contrib/bool_plperl/Makefile#L30
https://github.com/postgres/postgres/blob/master/src/pl/plpython/Makefile#L59-L72
https://github.com/postgres/postgres/blob/master/src/pl/tcl/Makefile#L35-L51
https://github.com/postgres/postgres/blob/master/config/python.m4#L62-L64x

There's plenty more, some of the more complicated cases are a bit less trivial
to search for.

Greetings,

Andres Freund

[1] https://postgr.es/m/CAGPVpCSKS9E0An4%3De7ZDnme%2By%3DWOcQFJYJegKO8kE9%3Dgh8NJKQ%40mail.gmail.com

  24. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-06-25T11:39:06Z

    Hi,

On 2024-06-25 11:54:56 +0100, Dave Page wrote:
> https://github.com/dpage/winpgbuild proves that the hacks above are not
> required *if* you build the dependencies in the recommended way for use
> with MSVC++ (where documented), otherwise just native Windows.

Partially it just means that some of the hacks are now located in the "build
dependencies" script.  E.g. you're renaming libintl.dll.a, libiconv.dll.a,
libuuid.a to something that's expected by the buildmethod. And the scripts
change the directory structure for several other dependencies (e.g. zstd, krb).


> If you, for example, build a dependency using Mingw/Msys, then you may get
> different filenames than if you build the same thing using its VC++
> solution or makefile. That's where most, if not all, of these issues come
> from.

Yes, that's precisely my point. The set of correct names / flags depends on
things outside of postgres control. Hence they should be handled outside of
postgres, not as part of postgres. Particularly because several of the
dependencies can be built in multiple ways, resulting in multiple library
names. And it doesn't even just differ by compiler, there's ways to get
different library names for some of the deps even with the same compiler!


> It's probably worth noting that "back in the day" when most of this stuff
> was built, there was no UCRT32 compiler option, and it really was a
> potential problem to mix VC++ and Mingw compiled binaries so there was a
> heavy focus on making sure everything was designed around the MSVC++ builds
> wherever they existed.

Agreed, this luckily got easier. But it also increased the variety of expected
library names / flags. It's entirely reasonable to build postgres with msvc
against an gcc built ICU or whatnot.

Greetings,

Andres Freund

  25. Re: Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-06-25T12:33:25Z

    Hi

On Tue, 25 Jun 2024 at 12:39, Andres Freund <andres@anarazel.de> wrote:

> Hi,
>
> On 2024-06-25 11:54:56 +0100, Dave Page wrote:
> > https://github.com/dpage/winpgbuild proves that the hacks above are not
> > required *if* you build the dependencies in the recommended way for use
> > with MSVC++ (where documented), otherwise just native Windows.
>
> Partially it just means that some of the hacks are now located in the
> "build
> dependencies" script.  E.g. you're renaming libintl.dll.a, libiconv.dll.a,
> libuuid.a to something that's expected by the buildmethod. And the scripts
> change the directory structure for several other dependencies (e.g. zstd,
> krb).
>
>
> > If you, for example, build a dependency using Mingw/Msys, then you may
> get
> > different filenames than if you build the same thing using its VC++
> > solution or makefile. That's where most, if not all, of these issues come
> > from.
>
> Yes, that's precisely my point. The set of correct names / flags depends on
> things outside of postgres control. Hence they should be handled outside of
> postgres, not as part of postgres. Particularly because several of the
> dependencies can be built in multiple ways, resulting in multiple library
> names. And it doesn't even just differ by compiler, there's ways to get
> different library names for some of the deps even with the same compiler!
>

Replying to this and your previous email.

I think we're in violent agreement here as to how things *should* be in an
ideal world. The issue for me is that this isn't an ideal world and the
current solution potentially makes it much harder to get a working
PostgreSQL build on Windows - not only that, but everyone doing those
builds potentially has to figure out how to get things to work for
themselves because we're pushing the knowledge outside of our build system.

I've been building Postgres on Windows for years (and like to think I'm
reasonably competent), and despite reading the docs I still ended up
starting multiple threads on the hackers list to try to understand how to
get v17 to build. Would we accept the current Meson setup on Linux if
people had to hand-craft .pc files, or install dependencies using multiple
third-party package managers to get it to work?

As I previously noted, I think we should default to pkgconfig/cmake
detection, but then fall back to what we did previously (with suitably
obnoxious warnings). Then at least a build environment that worked in the
past should work in the future.

-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

  26. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-07-09T06:51:01Z

    Hi,

I've been hacking on addressing some of the complaints (after having been
off-work in a somewhat unplanned way for most of the last two weeks).  With
some already opened and soon-to-be-proposed PRs to Dave's winbuild and the
attached changes I think the concerns can largely be addressed.


Here's the current set of changes:

0001: meson: Add missing argument to gssapi.h check
  Largely independent, but included to avoid conflicts

0002: Don't define HAVE_[GSSAPI_]GSSAPI_EXT_H
  Largely independent, but included to avoid conflicts

0003: meson: Add support for detecting gss without pkg-config
0004: meson: Add support for detecting ossp-uuid without pkg-config
  Do what it says on the tin. Neither includes dependency information via
  pkg-config or cmake in their upstream repos.

0005: meson: Add dependency lookups via names used by cmake

  This adds support for the alternative names used by cmake lookups. That
  addresses

0006: meson: nls: Handle intl requiring iconv
  This afaict is only required when dealing with a static libc, so it might be
  considered independent

0007: windows-tab-complete-workaround
  Just included so the build doesn't fail for me with all the dependencies
  installed.

0008: krb-vs-openssl-workaround
  Just included so the build doesn't fail for me with all the dependencies
  installed. There's a separate thread to discuss the right fix.

0009: wip: meson: reduce linker noise for executables

  This is mainly a minor quality of life thing for me when hacking on this. If
  a static libintl is used, link.exe outputs a message for every binary, which
  makes it harder to see warnings.

0010: meson: wip: tcl
  This is just some preliminary hacking needs more work.


Note that cmake is automatically installed as part of visual studio these
days.

Greetings,

Andres Freund

  27. Re: Meson far from ready on Windows

    Tristan Partin <tristan@partin.io> — 2024-07-16T20:53:45Z

    > From 9f7c96dfab4d807e668c9d32b44db5f4ff122e15 Mon Sep 17 00:00:00 2001
> From: Andres Freund <andres@anarazel.de>
> Date: Mon, 8 Jul 2024 15:55:56 -0700
> Subject: [PATCH v2 02/10] Don't define HAVE_[GSSAPI_]GSSAPI_EXT_H
> 
> The check for gssapi_ext.h was added in f7431bca8b0. As we require
> gssapi_ext.h to be present, there's no point in defining symbols for the
> header presence.
> 
> While at it, use cc.has_header() instead of cc.check_header(), that's a bit
> cheaper and it seems improbably that gssapi.h would compile while gssapi_ext.h
> would not.

improbable

Other than that, it looks pretty solid. Looks like we could help future 
us out by teaching compiler.find_library() to take a list of names to 
look at similar to how dependency() works now.

Reviewed-by: Tristan Partin <tristan@partin.io>

-- 
Tristan Partin
Neon (https://neon.tech)

  28. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-07-17T16:49:47Z

    Hi,

On 2024-07-16 15:53:45 -0500, Tristan Partin wrote:
> Other than that, it looks pretty solid.

Thanks for looking!  I'm thinking of pushing the first few patches soon-ish.

I'm debating between going for 17 + HEAD or also applying it to 16, to keep
the trees more similar.


> Looks like we could help future us out by teaching compiler.find_library()
> to take a list of names to look at similar to how dependency() works now.

Yep, that'd be useful.

Andres

  29. Re: Meson far from ready on Windows

    Andres Freund <andres@anarazel.de> — 2024-07-20T20:56:36Z

    Hi,

On 2024-07-17 09:49:47 -0700, Andres Freund wrote:
> On 2024-07-16 15:53:45 -0500, Tristan Partin wrote:
> > Other than that, it looks pretty solid.
> 
> Thanks for looking!  I'm thinking of pushing the first few patches soon-ish.
> 
> I'm debating between going for 17 + HEAD or also applying it to 16, to keep
> the trees more similar.

Pushed a number of these to 16 - HEAD.

Greetings,

Andres Freund

  30. Re: Meson far from ready on Windows

    Dave Page <dpage@pgadmin.org> — 2024-07-24T15:13:24Z

    On Sat, 20 Jul 2024 at 21:56, Andres Freund <andres@anarazel.de> wrote:

> Hi,
>
> On 2024-07-17 09:49:47 -0700, Andres Freund wrote:
> > On 2024-07-16 15:53:45 -0500, Tristan Partin wrote:
> > > Other than that, it looks pretty solid.
> >
> > Thanks for looking!  I'm thinking of pushing the first few patches
> soon-ish.
> >
> > I'm debating between going for 17 + HEAD or also applying it to 16, to
> keep
> > the trees more similar.
>
> Pushed a number of these to 16 - HEAD.
>

Thanks. I've updated winpgbuild with the additional pkgconfig file needed
for ICU now, so it should better match a *nix build.

Any chance you can look at the GSSAPI/OpenSSL X509_NAME conflict one? I'm
still having to patch around that to build with all the dependencies.

https://www.postgresql.org/message-id/flat/CA%2BOCxoxwsgi8QdzN8A0OPGuGfu_1vEW3ufVBnbwd3gfawVpsXw%40mail.gmail.com

-- 
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com

PGDay UK 2024, 11th September, London: https://2024.pgday.uk/

  31. Re: Meson far from ready on Windows

    Wolfgang Walther <walther@technowledgy.de> — 2024-08-18T14:30:11Z

    Andres Freund:
>> That's not necessarily true. The nix package manager and thus NixOS track all dependencies for a piece of software. If any of the dependencies are updated, all dependents are rebuilt, too. So the security concern doesn't apply here. There is a "static overlay", which builds everything linked fully statically.
> 
> Right. There's definitely some scenario where it's ok, I was simplifying a bit.
> 
>> Unfortunately, PostgreSQL doesn't build in that, so far.
> 
> I've built mostly statically linked pg without much of a problem, what trouble did you encounter? Think there were some issues with linking Kerberos and openldap statically, but not on postgres' side.

Mostly the "can't disable shared libraries / backend builds" part 
mentioned below.

> Building the postgres backend without support for dynamic linking doesn't make sense though. Extensions are just stop ingrained part of pg.

I think there might be some limited use-cases for a fully-static 
postgres backend without the ability to load extensions: Even if we get 
libpq to build fine in the fully-static overlay mentioned above, a lot 
of reverse dependencies have to disable tests, because they need a 
running postgres server to run their tests against.

Providing a really simple postgres backend, with only minimal 
functionality, would allow some basic sanity tests, even in this purely 
static environment.

>> Lately, I have been looking into building at least libpq in that static overlay, via Meson. There are two related config options:
>> -Ddefault_library=shared|static|both
>> -Dprefer_static
>>
>> The first controls which libraries (libpq, ...) to build ourselves. The second controls linking, IIUC also against external dependencies.
> 
> Pg by default builds a static libpq on nearly all platforms (not aix I think and maybe not Windows when building with autoconf, not sure about the old msvc system) today?

Yes, PG builds a static libpq today. But it's hard-to-impossible to 
*disable building the shared library*. In the fully static overlay, this 
causes the build to fail, because shared libraries can't be build.

>> Maybe it would be a first step to support -Dprefer_static?
> 
> That should work for nearly all dependencies today. Except for libintl, I think.  I found that there are a lot of buglets in static link dependencies of various libraries though.

To support prefer_static, we'd also have to look at our internal 
linking, i.e. whether for example psql is linked against libpq 
statically or dynamically. Once prefer_static controls that, that's 
already a step forward to be able to build more of the code-base without 
shared libraries available.

Best,

Wolfgang