Horton

[RFC] building postgres with meson

Andres Freund <andres@anarazel.de>

View thread View on postgr.es
From: Andres Freund <andres@anarazel.de>
To: pgsql-hackers@postgresql.org
Date: 2021-10-12T08:37:21Z
Lists: pgsql-hackers

Attachments

Hi,

For the last year or so I've on and off tinkered with $subject.  I think
it's in a state worth sharing now.  First, let's look at a little
comparison.

My workstation:

non-cached configure:
current:        11.80s
meson:           6.67s

non-cached build (world-bin):
current:        40.46s
ninja:           7.31s

no-change build:
current:         1.17s
ninja:           0.06s

test world:
current:          105s
meson:             63s


What actually started to motivate me however were the long times windows
builds took to come back with testsresults. On CI, with the same machine
config:

build:
current:          202s (doesn't include genbki etc)
meson+ninja:      140s
meson+msbuild:    206s


test:
current:         1323s (many commands)
meson:            903s (single command)

(note that the test comparison isn't quite fair - there's a few tests
missing, but it's just small contrib ones afaik)


The biggest difference to me however is not the speed, but how readable
the output is.

Running the tests with meson in a terminal, shows the number of tests
that completed out of how many total, how much time has passed, how long
the currently running tests already have been running.

At the end of a testrun a count of tests is shown:

188/189 postgresql:tap+pg_basebackup / pg_basebackup/t/010_pg_basebackup.pl                         OK               39.51s   110 subtests passed
189/189 postgresql:isolation+snapshot_too_old / snapshot_too_old/isolation                          OK               62.93s


Ok:                 188
Expected Fail:      0
Fail:               1
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /tmp/meson/meson-logs/testlog.txt


The log has the output of the tests and ends with:

Summary of Failures:
120/189 postgresql:tap+recovery / recovery/t/007_sync_rep.pl                               ERROR             7.16s   (exit status 255 or signal 127 SIGinvalid)


Quite the difference to make check-world -jnn output.


So, now that the teasing is done, let me explain a bit what lead me down
this path:

Autoconf + make is not being actively developed. Especially autoconf is
*barely* in maintenance mode - despite many shortcomings and bugs. It's
also technology that very few want to use - autoconf m4 is scary, and
it's scarier for people that started more recently than a lot of us
committers for example.

Recursive make as we use it is hard to get right. One reason the clean
make build is so slow compared to meson is that we had to resort to
.NOTPARALLEL to handle dependencies in a bunch of places. And despite
that, I quite regularly see incremental build failures that can be
resolved by retrying the build.

While we have incremental build via --enable-depend, they don't work
that reliable (i.e. misses necessary rebuilds) and yet is often too
aggressive. More modern build system can keep track of the precise
command used to build a target and rebuild it when that command changes.


We also don't just have the autoconf / make buildsystem, there's also
the msvc project generator - something most of us unix-y folks do not
like to touch. I think that, combined with there being no easy way to
run all tests, and it being just different, really hurt our windows
developer appeal (and subsequently the quality of postgres on
windows). I'm not saying this to ding the project generator - that was
well before there were decent "meta" buildsystems out there (and in some
ways it is a small one itself).


The last big issue I have with the current situation is that there's no
good test integration. make check-world output is essentially unreadable
/ not automatically parseable. Which led to the buildfarm having a
separate list of things it needs to test, so that failures can be
pinpointed and paired with appropriate logs. That approach unfortunately
doesn't scale well to multi-core CPUs, slowing down the buildfarm by a
fair bit.


This all led to me to experiment with improvements. I tried a few
somewhat crazy but incremental things like converting our buildsystem to
non-recursive make (I got it to build the backend, but it's too hard to
do manually I think), or to not run tests during the recursive make
check-world, but to append commands to a list of tests, that then is run
by a helper (can kinda be made to work).  In the end I concluded that
the amount of time we'd need to invest to maintain our more-and-more
custom buildsystem going forward doesn't make sense.


Which lead me to look around and analyze which other buildsystems there
are that could make some sense for us. The halfway decent list includes,
I think:
1) cmake
2) bazel
3) meson


cmake would be a decent choice, I think. However, I just can't fully
warm up to it. Something about it just doesn't quite sit right with
me. That's not a good enough reason to prevent others from suggesting to
use it, but it's good enough to justify not investing a lot of time in
it myself.

Bazel has some nice architectural properties. But it requires a JVM to
run - I think that basically makes it insuitable for us. And the build
information seems quite arduous to maintain too.

Which left me with meson. It is a meta-buildsystem that can do the
actual work of building via ninja (the most common one, also targeted by
cmake), msbuild (visual studio project files, important for GUI work)
and xcode projects (I assume that's for a macos IDE, but I haven't tried
to use it). Meson roughly does what autoconf+automake did, in a
python-esque DSL, and outputs build-instructions for ninja / msbuild /
xcode.  One interesting bit is that meson itself is written in python (
and fairly easy to contribute too - I got a few changes in now).


I don't think meson is perfect architecturally - e.g. its insistence on
not having functions ends up making it a bit harder to not end up
duplicating code. There's some user-interface oddities that are now hard
to fix fully, due to the faily wide usage. But all-in-all it's pretty
nice to use.


Its worth calling out that a lot of large open source projects have been
/ are migrating to meson. qemu/kvm, mesa (core part of graphics stack on
linux and also widely used in other platforms), a good chunk of GNOME,
and quite a few more. Due to that it seems unlikely to be abandoned
soon.


As far as I can tell the only OS that postgres currently supports that
meson doesn't support is HPUX. It'd likely be fairly easy to add
gcc-on-hpux support, a chunk more to add support for the proprietary
ones.


The attached patch (meson support is 0016, the rest is prerequisites
that aren't that interesting at this stage) converts most of postgres to
meson. There's a few missing contrib modules, only about half the
optional library dependencies are implemented, and I've only built on
x64. It builds on freebsd, linux, macos and windows (both ninja and
msbuild) and cross builds from linux to windows. Thomas helped make the
freebsd / macos pieces a reality, thanks!

I took a number of shortcuts (although there used to be a *lot*
more). So this shouldn't be reviewed to the normal standard of the
community - it's a prototype.  But I think it's in a complete enough
shape that it allows to do a well-informed evaluation.

What doesn't yet work/ build:

- plenty optional libraries, contrib, NLS, docs build

- PGXS - and I don't yet know what to best do about it. One
  backward-compatible way would be to continue use makefiles for pgxs,
  but do the necessary replacement of Makefile.global.in via meson (and
  not use that for postgres' own build).  But that doesn't really
  provide a nicer path for building postgres extensions on windows, so
  it'd definitely not be a long-term path.

- JIT bitcode generation for anything but src/backend.

- anything but modern-ish x86. That's proably a small amount of work,
  but something that needs to be done.

- exporting all symbols for extension modules on windows (the stuff for
  postgres is implemented). Instead I marked the relevant symbols als
  declspec(dllexport). I think we should do that regardless of the
  buildsystem change. Restricting symbol visibility via gcc's
  -fvisibility=hidden for extensions results in a substantially reduced
  number of exported symbols, and even reduces object size (and I think
  improves the code too).  I'll send an email about that separately.




There's a lot more stuff to talk about, but I'll stop with a small bit
of instructions below:


Demo / instructions:
# Get code
git remote add andres git@github.com:anarazel/postgres.git
git fetch andres
git checkout --track andres/meson

# setup build directory
meson setup build --buildtype debug
cd build

# build (uses automatically as many cores as available)
ninja

# change configuration, build again
meson configure -Dssl=openssl
ninja

# run all tests
meson test

# run just recovery tests
meson test --suite setup --suite recovery

# list tests
meson test --list


Greetings,

Andres Freund

Commits

  1. meson: Add 'running' test setup, as a replacement for installcheck

  2. meson: Add support for building with precompiled headers

  3. windows: Adjust FD_SETSIZE via commandline define

  4. meson: docs: Add xml{lint,proc} wrapper to collect dependencies

  5. meson: ecpg: Split definition of static and shared libraries

  6. windows: Set UMDF_USING_NTSTATUS globally, include ntstatus.h

  7. ci: Add hint about downloadable logs to README

  8. meson: Set up absolute rpaths to libdir

  9. meson: Include CFLAGS/c_args in summary and pg_config output

  10. windows: remove date from version number in win32ver.rc

  11. meson: Add initial version of meson based build system

  12. ci: windows: set error mode to not include SEM_NOGPFAULTERRORBOX

  13. Refactor PG_TEST_EXTRA logic in autoconf build

  14. Split TESTDIR into TESTLOGDIR and TESTDATADIR

  15. Don't hardcode tmp_check/ as test directory for tap tests

  16. Extend gendef.pl in preparation for meson

  17. Include c.h instead of postgres.h in src/port/*p{read,write}*.c

  18. Remove DLLTOOL, DLLWRAP from configure / Makefile.global.in

  19. Run xmllint validation only once

  20. Bump minimum Perl version to 5.14

  21. Move gramparse.h to src/backend/parser

  22. Run perltidy over Catalog.pm

  23. Parse catalog .dat files as a whole when compiling the backend

  24. Build all Flex files standalone

  25. Move private declarations shared between guc.c and guc-file.l to new header

  26. Preparatory refactoring for compiling guc-file.c standalone

  27. Move darwin sysroot determination into separate file

  28. Fix MSVC warning in compat_informix/rnull.pgc

  29. solaris: Use versioning scripts instead of -Bsymbolic

  30. Change shared library installation naming on macOS

  31. regress: allow to specify directory containing expected files, for ecpg

  32. Don't add HAVE_LDAP_H HAVE_WINLDAP_H to pg_config.h

  33. Refactor dtrace postprocessing make rules

  34. Add output directory option to gen_node_support.pl

  35. Add output directory argument to generate-unicode_norm_table.pl

  36. Add output file argument to generate-errcodes.pl

  37. Add output path arg in generate-lwlocknames.pl

  38. Move snowball_create.sql creation into perl file

  39. ecpg: Output dir, source dir, stamp file argument for preproc/*.pl

  40. psql: Output dir and dependency generation for sql_help

  41. Deal with paths containing \ and spaces in basebackup_to_shell tests

  42. Remove LLVM_CONFIG from Makefile.global.in

  43. Make update-unicode target work in vpath builds

  44. Refactor DLSUFFIX handling

  45. ldap tests: Add paths for openbsd.

  46. ldap tests: Don't run on unsupported operating systems.

  47. Remove check for accept() argument types

  48. Move our p{read,write}v replacements into their own files.

  49. Adjust yywrap macro for non-reentrant scanners for MSVC.

  50. Remove any -arch switches given in ExtUtils::Embed's ldopts from our

  51. Change PL/Perl and Pg interface build to use configured compiler and

  52. Apparently, on some systems, ExtUtils::Embed and MakeMaker are slightly