Thread
-
meson vs. llvm bitcode files
Peter Eisentraut <peter@eisentraut.org> — 2024-09-05T08:56:26Z
The meson build currently does not produce llvm bitcode (.bc) files. AFAIK, this is the last major regression for using meson for production builds. Is anyone working on that? I vaguely recall that some in-progress code was shared a couple of years ago, but I haven't seen anything since. It would be great if we could collect any existing code and notes to maybe get this moving again.
-
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2024-09-05T09:24:51Z
Hi, On Thu, 5 Sept 2024 at 11:56, Peter Eisentraut <peter@eisentraut.org> wrote: > > The meson build currently does not produce llvm bitcode (.bc) files. > AFAIK, this is the last major regression for using meson for production > builds. > > Is anyone working on that? I vaguely recall that some in-progress code > was shared a couple of years ago, but I haven't seen anything since. It > would be great if we could collect any existing code and notes to maybe > get this moving again. I found that Andres shared a patch (v17-0021-meson-Add-LLVM-bitcode-emission.patch) a while ago [1]. [1] https://www.postgresql.org/message-id/20220927011951.j3h4o7n6bhf7dwau%40awork3.anarazel.de -- Regards, Nazir Bilal Yavuz Microsoft
-
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2025-03-07T10:52:09Z
Hi, On Thu, 5 Sept 2024 at 12:24, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > I found that Andres shared a patch > (v17-0021-meson-Add-LLVM-bitcode-emission.patch) a while ago [1]. Andres and I continued to work on that. I think the patches are in sharable state now and I wanted to hear opinions before proceeding further. After applying the patches, bitcode files should be installed into $pkglibdir/bitcode/ directory if the llvm is found. There are 6 patches attached: v1-0001-meson-Add-generated-header-stamps: This patch is trivial. Instead of having targets depending directly on the generated headers, have them depend on a stamp file. The benefit of using a stamp file is that it makes ninja.build smaller and meson setup faster. ---------- v1-0002-meson-Add-postgresql-extension.pc-for-building-extension-libraries: This patch is for generating postgresql-extension.pc file which can be used for building extensions libraries. Normally, there is no need to use this .pc file for generating bitcode files. However, since there is no clear way to get all include paths for building bitcode files, this .pc file is later used for this purpose (by running pkg-config --cflags-only-I postgresql-extension-uninstalled.pc) [1]. ---------- v1-0003-meson-Test-building-extensions-by-using-postgresql-extension.pc: [Not needed for generating bitcode files] This is a patch for testing if extensions can be built by using postgresql-extension.pc. I added that commit as an example of using postgresql-extension.pc to build extensions. ---------- v1-0004-meson-WIP-Add-docs-for-postgresql-extension.pc: [Not needed for generating bitcode files] I added this patch in case we recommend people to use postgresql-extension.pc to build extension libraries. I am not sure if we want to do that because there are still TODOs about postgresql-extension.pc like running test suites. I just wanted to show my plan, dividing 'Extension Building Infrastructure' into two, 'PGXS' and 'postgresql-extension.pc'. ---------- v1-0005-meson-Add-LLVM-bitcode-emission: This patch adds required infrastructure to generate bitcode files and uses postgresql-extension-uninstalled.pc to get include paths for generating bitcode files [1]. ---------- v1-0006-meson-Generate-bitcode-files-of-contrib-extension.patch: This patch adds manually selected contrib libraries to generate their bitcode files. These libraries are selected manually, depending on - If they have SQL callable functions - If the library functions are short enough (the performance gain from bitcode files is too minimal compared to the function's run time, so this type of libraries are omitted). Any kind of feedback would be appreciated. -- Regards, Nazir Bilal Yavuz Microsoft
-
Re: meson vs. llvm bitcode files
Diego Fronza <diego.fronza@percona.com> — 2025-03-10T22:04:19Z
Hello, I did a full review on the provided patches plus some tests, I was able to validate that the loading of bitcode modules is working also JIT works for both backend and contrib modules. To test JIT on contrib modules I just lowered the costs for all jit settings and used the intarray extension, using the data/test__int.data: CREATE EXTENSION intarray; CREATE TABLE test__int( a int[] );1 \copy test__int from 'data/test__int.data' For queries any from line 98+ on contrib/intarray/sql/_int.sql will work. Then I added extra debug messages to llvmjit_inline.cpp on add_module_to_inline_search_path() function, also on llvm_build_inline_plan(), I was able to see many functions in this module being successfully inlined. I'm attaching a new patch based on your original work which add further support for generating bitcode from: - Generated backend sources: processed by flex, bison, etc. - Generated contrib module sources, On this patch I just included fmgrtab.c and src/backend/parser for the backend generated code. For contrib generated sources I added contrib/cube as an example. All relevant details about the changes are included in the patch itself. As you may know already I also created a PR focused on llvm bitcode emission on meson, it generates bitcode for all backend and contribution modules, currently under review by some colleagues at Percona: https://github.com/percona/postgres/pull/103 I'm curious if we should get all or some of the generated backend sources compiled to bitcode, similar to contrib modules. Please let me know your thoughts and how we can proceed to get this feature included, thank you. Regards, Diego Fronza Percona On Fri, Mar 7, 2025 at 7:52 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > Hi, > > On Thu, 5 Sept 2024 at 12:24, Nazir Bilal Yavuz <byavuz81@gmail.com> > wrote: > > > > I found that Andres shared a patch > > (v17-0021-meson-Add-LLVM-bitcode-emission.patch) a while ago [1]. > > Andres and I continued to work on that. I think the patches are in > sharable state now and I wanted to hear opinions before proceeding > further. After applying the patches, bitcode files should be installed > into $pkglibdir/bitcode/ directory if the llvm is found. > > There are 6 patches attached: > > v1-0001-meson-Add-generated-header-stamps: > > This patch is trivial. Instead of having targets depending directly on > the generated headers, have them depend on a stamp file. The benefit > of using a stamp file is that it makes ninja.build smaller and meson > setup faster. > ---------- > > v1-0002-meson-Add-postgresql-extension.pc-for-building-extension-libraries: > > This patch is for generating postgresql-extension.pc file which can be > used for building extensions libraries. > > Normally, there is no need to use this .pc file for generating bitcode > files. However, since there is no clear way to get all include paths > for building bitcode files, this .pc file is later used for this > purpose (by running pkg-config --cflags-only-I > postgresql-extension-uninstalled.pc) [1]. > ---------- > > v1-0003-meson-Test-building-extensions-by-using-postgresql-extension.pc: > [Not needed for generating bitcode files] > > This is a patch for testing if extensions can be built by using > postgresql-extension.pc. I added that commit as an example of using > postgresql-extension.pc to build extensions. > ---------- > > v1-0004-meson-WIP-Add-docs-for-postgresql-extension.pc: [Not needed > for generating bitcode files] > > I added this patch in case we recommend people to use > postgresql-extension.pc to build extension libraries. I am not sure if > we want to do that because there are still TODOs about > postgresql-extension.pc like running test suites. I just wanted to > show my plan, dividing 'Extension Building Infrastructure' into two, > 'PGXS' and 'postgresql-extension.pc'. > ---------- > > v1-0005-meson-Add-LLVM-bitcode-emission: > > This patch adds required infrastructure to generate bitcode files and > uses postgresql-extension-uninstalled.pc to get include paths for > generating bitcode files [1]. > ---------- > > v1-0006-meson-Generate-bitcode-files-of-contrib-extension.patch: > > This patch adds manually selected contrib libraries to generate their > bitcode files. These libraries are selected manually, depending on > - If they have SQL callable functions > - If the library functions are short enough (the performance gain from > bitcode files is too minimal compared to the function's run time, so > this type of libraries are omitted). > > Any kind of feedback would be appreciated. > > -- > Regards, > Nazir Bilal Yavuz > Microsoft >
-
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2025-03-12T10:26:46Z
Hi, On Tue, 11 Mar 2025 at 01:04, Diego Fronza <diego.fronza@percona.com> wrote: > I did a full review on the provided patches plus some tests, I was able to validate that the loading of bitcode modules is working also JIT works for both backend and contrib modules. Thank you! > To test JIT on contrib modules I just lowered the costs for all jit settings and used the intarray extension, using the data/test__int.data: > CREATE EXTENSION intarray; > CREATE TABLE test__int( a int[] );1 > \copy test__int from 'data/test__int.data' > > For queries any from line 98+ on contrib/intarray/sql/_int.sql will work. > > Then I added extra debug messages to llvmjit_inline.cpp on add_module_to_inline_search_path() function, also on llvm_build_inline_plan(), I was able to see many functions in this module being successfully inlined. > > I'm attaching a new patch based on your original work which add further support for generating bitcode from: Thanks for doing that! > - Generated backend sources: processed by flex, bison, etc. > - Generated contrib module sources, I think we do not need to separate these two. foreach srcfile : bitcode_module['srcfiles'] - if meson.version().version_compare('>=0.59') + srcfilename = '@0@'.format(srcfile) + if srcfilename.startswith('<CustomTarget') + srcfilename = srcfile.full_path().split(meson.build_root() + '/')[1] + elif meson.version().version_compare('>=0.59') Also, checking if the string starts with '<CustomTarget' is a bit hacky, and 'srcfilename = '@0@'.format(srcfile)' causes a deprecation warning. So, instead of this we can process all generated sources like how generated backend sources are processed. I updated the patch with that. > On this patch I just included fmgrtab.c and src/backend/parser for the backend generated code. > For contrib generated sources I added contrib/cube as an example. I applied your contrib/cube example and did the same thing for the contrib/seg. > All relevant details about the changes are included in the patch itself. > > As you may know already I also created a PR focused on llvm bitcode emission on meson, it generates bitcode for all backend and contribution modules, currently under review by some colleagues at Percona: https://github.com/percona/postgres/pull/103 > I'm curious if we should get all or some of the generated backend sources compiled to bitcode, similar to contrib modules. I think we can do this. I added other backend sources like you did in the PR but attached it as another patch (0007) because I wanted to hear other people's opinions on that first. v3 is attached. -- Regards, Nazir Bilal Yavuz Microsoft -
Re: meson vs. llvm bitcode files
Diego Fronza <diego.fronza@percona.com> — 2025-03-12T13:39:19Z
Hi, The v7 patch looks good to me, handling the bitcode modules in a uniform way and also avoiding the hacky code and warnings, much better now. A small note about the bitcode emission for generated sources in contrib, using cube as example, currently it creates two dict entries in a list: bc_seg_gen_sources = [{'srcfiles': [seg_scan]}] bc_seg_gen_sources += {'srcfiles': [seg_parse[0]]} Then pass it to the bitcode_modules: bitcode_modules += { ... 'gen_srcfiles': bc_seg_gen_sources, } It could be passed as a list with a single dict, since both generated sources share the same compilation flags: bitcode_modules += { ... 'gen_srcfiles': [ { 'srcfiles': [cube_scan, cube_parse[0]] }. ] } Both approaches work, the first one has the advantage of being able to pass separate additional_flags per generated source. Thanks for your reply Nazir, also waiting for more opinions on this. Regards, Diego On Wed, Mar 12, 2025 at 7:27 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > Hi, > > On Tue, 11 Mar 2025 at 01:04, Diego Fronza <diego.fronza@percona.com> > wrote: > > I did a full review on the provided patches plus some tests, I was able > to validate that the loading of bitcode modules is working also JIT works > for both backend and contrib modules. > > Thank you! > > > To test JIT on contrib modules I just lowered the costs for all jit > settings and used the intarray extension, using the data/test__int.data: > > CREATE EXTENSION intarray; > > CREATE TABLE test__int( a int[] );1 > > \copy test__int from 'data/test__int.data' > > > > For queries any from line 98+ on contrib/intarray/sql/_int.sql will work. > > > > Then I added extra debug messages to llvmjit_inline.cpp on > add_module_to_inline_search_path() function, also on > llvm_build_inline_plan(), I was able to see many functions in this module > being successfully inlined. > > > > I'm attaching a new patch based on your original work which add further > support for generating bitcode from: > > Thanks for doing that! > > > - Generated backend sources: processed by flex, bison, etc. > > - Generated contrib module sources, > > I think we do not need to separate these two. > > foreach srcfile : bitcode_module['srcfiles'] > - if meson.version().version_compare('>=0.59') > + srcfilename = '@0@'.format(srcfile) > + if srcfilename.startswith('<CustomTarget') > + srcfilename = srcfile.full_path().split(meson.build_root() + '/')[1] > + elif meson.version().version_compare('>=0.59') > > Also, checking if the string starts with '<CustomTarget' is a bit > hacky, and 'srcfilename = '@0@'.format(srcfile)' causes a deprecation > warning. So, instead of this we can process all generated sources like > how generated backend sources are processed. I updated the patch with > that. > > > On this patch I just included fmgrtab.c and src/backend/parser for the > backend generated code. > > For contrib generated sources I added contrib/cube as an example. > > I applied your contrib/cube example and did the same thing for the > contrib/seg. > > > All relevant details about the changes are included in the patch itself. > > > > As you may know already I also created a PR focused on llvm bitcode > emission on meson, it generates bitcode for all backend and contribution > modules, currently under review by some colleagues at Percona: > https://github.com/percona/postgres/pull/103 > > I'm curious if we should get all or some of the generated backend > sources compiled to bitcode, similar to contrib modules. > > I think we can do this. I added other backend sources like you did in > the PR but attached it as another patch (0007) because I wanted to > hear other people's opinions on that first. > > v3 is attached. > > -- > Regards, > Nazir Bilal Yavuz > Microsoft > -
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2025-03-13T10:11:44Z
Hi, On Wed, 12 Mar 2025 at 16:39, Diego Fronza <diego.fronza@percona.com> wrote: > > Hi, > > The v7 patch looks good to me, handling the bitcode modules in a uniform way and also avoiding the hacky code and warnings, much better now. > > A small note about the bitcode emission for generated sources in contrib, using cube as example, currently it creates two dict entries in a list: > bc_seg_gen_sources = [{'srcfiles': [seg_scan]}] > bc_seg_gen_sources += {'srcfiles': [seg_parse[0]]} > > Then pass it to the bitcode_modules: > bitcode_modules += { > ... > 'gen_srcfiles': bc_seg_gen_sources, > } > > It could be passed as a list with a single dict, since both generated sources share the same compilation flags: > bitcode_modules += { > ... > 'gen_srcfiles': [ > { 'srcfiles': [cube_scan, cube_parse[0]] }. > ] > } > > Both approaches work, the first one has the advantage of being able to pass separate additional_flags per generated source. I liked the current approach as it makes bitcode_modules easier to understand but both approaches work for me as well. One thing I noticed is that gen_srcfiles['srcfiles'] seems wrong. gen_sources is a better name compared to gen_srcfiles. So, I changed it to gen_sources in v4. -- Regards, Nazir Bilal Yavuz Microsoft -
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2025-04-29T08:23:54Z
Hi, On Thu, 13 Mar 2025 at 13:11, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > One thing I noticed is that gen_srcfiles['srcfiles'] seems wrong. > gen_sources is a better name compared to gen_srcfiles. So, I changed > it to gen_sources in v4. Rebase is needed due to b1720fe63f, v5 is attached. -- Regards, Nazir Bilal Yavuz Microsoft
-
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2025-07-07T08:45:06Z
Hi, On Tue, 29 Apr 2025 at 11:23, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > Hi, > > On Thu, 13 Mar 2025 at 13:11, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > One thing I noticed is that gen_srcfiles['srcfiles'] seems wrong. > > gen_sources is a better name compared to gen_srcfiles. So, I changed > > it to gen_sources in v4. > > Rebase is needed due to b1720fe63f, v5 is attached. Mandatory rebase, v6 is attached. -- Regards, Nazir Bilal Yavuz Microsoft
-
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2025-08-13T13:25:27Z
Hi, On Mon, 7 Jul 2025 at 11:45, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > Mandatory rebase, v6 is attached. Rebase is needed due to 01d6832c10, v7 is attached. -- Regards, Nazir Bilal Yavuz Microsoft
-
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2025-10-31T12:13:13Z
Hi, On Wed, 13 Aug 2025 at 16:25, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > Hi, > > On Mon, 7 Jul 2025 at 11:45, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > > > Mandatory rebase, v6 is attached. > > Rebase is needed due to 01d6832c10, v7 is attached. Rebase is needed due to 16607718c0, v8 is attached. -- Regards, Nazir Bilal Yavuz Microsoft
-
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-01-16T11:33:31Z
Hi, On Fri, 31 Oct 2025 at 15:13, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > On Wed, 13 Aug 2025 at 16:25, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > > > Hi, > > > > On Mon, 7 Jul 2025 at 11:45, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > > > > > Mandatory rebase, v6 is attached. > > > > Rebase is needed due to 01d6832c10, v7 is attached. > > Rebase is needed due to 16607718c0, v8 is attached. Rebase is needed. Also, there is small functional change in 0002: def remove_duplicates(duplicate_str): - words = duplicate_str.split() + # Remove duplicates based on basename as there could be a mix of both full + # paths and bare binary names. + words = [os.path.basename(word) for word in duplicate_str.split()] return ' '.join(sorted(set(words), key=words.index)) It is because MacOS was failing due to there being 2 instances of ccache, one is with full path '/opt/local/bin/ccache' and one is just the binary name 'ccache'. remove_duplicates() function did not remove them as it compared full strings before, now it compares only basenames. -- Regards, Nazir Bilal Yavuz Microsoft -
Re: meson vs. llvm bitcode files
Peter Eisentraut <peter@eisentraut.org> — 2026-03-12T10:54:11Z
On 16.01.26 12:33, Nazir Bilal Yavuz wrote: > Hi, > > On Fri, 31 Oct 2025 at 15:13, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: >> >> On Wed, 13 Aug 2025 at 16:25, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: >>> >>> Hi, >>> >>> On Mon, 7 Jul 2025 at 11:45, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: >>>> >>>> Mandatory rebase, v6 is attached. >>> >>> Rebase is needed due to 01d6832c10, v7 is attached. >> >> Rebase is needed due to 16607718c0, v8 is attached. > > Rebase is needed. Also, there is small functional change in 0002: > > def remove_duplicates(duplicate_str): > - words = duplicate_str.split() > + # Remove duplicates based on basename as there could be a mix of both full > + # paths and bare binary names. > + words = [os.path.basename(word) for word in duplicate_str.split()] > return ' '.join(sorted(set(words), key=words.index)) > > It is because MacOS was failing due to there being 2 instances of > ccache, one is with full path '/opt/local/bin/ccache' and one is just > the binary name 'ccache'. remove_duplicates() function did not remove > them as it compared full strings before, now it compares only > basenames. Some review comments from me. v9-0001-meson-Add-postgresql-extension.pc-for-building-ex.patch Need to think about whether "extension" is the correct term. New meson message: NOTICE: Future-deprecated features used: * 0.62.0: {'pkgconfig.generate variable for builtin directories'} The comment that introduces postgresql-extension-warnings.pc says +# Extension modules should likely also use -fwrapv etc. But it it's a bit odd +# to expose it to a .pc file? but then -fwrapv ends up in postgresql-extension.pc anyway. Not sure what was intended here. Also, the description "PostgreSQL Extension Support - Compiler Warnings" could be clarified, like "with recommended compiler warnings" or "with compiler warnings same as core code" or similar. The Requires list in my case is for example Requires: krb5-gssapi, icu-uc, icu-i18n, ldap, libxml-2.0 >= 2.6.23, liblz4, openssl, zlib, libzstd >= 1.4.0 but I don't think these are actually required for building extensions (unless a particular extension directly makes use of one of them, in which case they should declare that on their own). If we are going to install these .pc files, we also need to build them with with makefiles. Alternatively, we could not install them for now and just use them internally. v9-0002-meson-Test-building-extensions-by-using-postgresq.patch Not sure if this was meant to be kept or it's just for local testing. New meson warnings: WARNING: Deprecated features used: * 0.55.0: {'ExternalProgram.path'} * 0.56.0: {'meson.build_root'} src/test/modules/ seems like the wrong location, since it's not a module or a test module. I don't know if it's possible to make meson use a different file than meson.build, but if so, it might be better to keep these test meson.build files together with their extensions, like contrib/amcheck/meson-test.build. Similar to how we have "PGXS" build support in the makefiles. Otherwise, I'm afraid this will get annoying and error-prone if one has to remember to update other files under src/test/ when adding for example a new .sql file to amcheck. Also, the driver script is at 'src/tools/ci/test_meson_extensions', but you are using it outside of CI, so that's not a good location. v9-0003-meson-WIP-Add-docs-for-postgresql-extension.pc.patch Let's not rename existing ids. It seems to me that the .pc file can also be used without meson. Let's take that into account a bit. For example, the id="extend-postgres-meson" could be id="extend-postgres-pkg-config" or similar. Your text ends with a colon. Did you mean to add more text? Maybe an example meson.build would be good. v9-0005-meson-Add-LLVM-bitcode-emissions-for-contrib-libr.patch +# some libraries include "hstore/hstore.h" instead of "hstore.h" It seems to me that the former is correct, but if not then we should fix it. -
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-03-18T12:44:18Z
Hi, On Thu, 12 Mar 2026 at 13:54, Peter Eisentraut <peter@eisentraut.org> wrote: > > Some review comments from me. Thank you for looking into this! > v9-0001-meson-Add-postgresql-extension.pc-for-building-ex.patch > > Need to think about whether "extension" is the correct term. It looks correct to me. Do you have any suggestions? > New meson message: > > NOTICE: Future-deprecated features used: > * 0.62.0: {'pkgconfig.generate variable for builtin directories'} Fixed. > The comment that introduces postgresql-extension-warnings.pc says > > +# Extension modules should likely also use -fwrapv etc. But it it's a > bit odd > +# to expose it to a .pc file? > > but then -fwrapv ends up in postgresql-extension.pc anyway. Not sure > what was intended here. I asked Andres off-list and Andres said that we need to have these flags inside the .pc file but it is not very nice since these flags (-fwrapv for example) change the behavior. Maybe Andres could clarify this better. > Also, the description "PostgreSQL Extension Support - Compiler > Warnings" could be clarified, like "with recommended compiler > warnings" or "with compiler warnings same as core code" or similar. Done. I changed it to "PostgreSQL Extension Support with compiler warnings the same as core code". I am not sure about uppercase/lowercase but it seems okay to me. > The Requires list in my case is for example > > Requires: krb5-gssapi, icu-uc, icu-i18n, ldap, libxml-2.0 >= 2.6.23, > liblz4, openssl, zlib, libzstd >= 1.4.0 > > but I don't think these are actually required for building extensions > (unless a particular extension directly makes use of one of them, in > which case they should declare that on their own). It seems that is how meson pkgconfig.generate() handles the dependencies, please see [1]: ... * Dependencies provided by pkg-config are added into Requires: or Requires.private:. If a version was specified when declaring that dependency it will be written into the generated file too. ... > If we are going to install these .pc files, we also need to build them > with with makefiles. Alternatively, we could not install them for now > and just use them internally. Unfortunately, these .pc files are always installed in meson build. I added a WIP patch (0007) for building .pc files with makefiles, I am not sure if I am following the correct way. I would appreciate any help on this. > v9-0002-meson-Test-building-extensions-by-using-postgresq.patch > > Not sure if this was meant to be kept or it's just for local testing. I think we can have it in Postgres, it shows that generated .pc files work and extensions can be built by using these .pc files and using meson build. But maybe we can build one extension instead of three (or a dummy extension), what do you think? > New meson warnings: > > WARNING: Deprecated features used: > * 0.55.0: {'ExternalProgram.path'} > * 0.56.0: {'meson.build_root'} Fixed. > src/test/modules/ seems like the wrong location, since it's not a > module or a test module. > > I don't know if it's possible to make meson use a different file than > meson.build, but if so, it might be better to keep these test > meson.build files together with their extensions, like > contrib/amcheck/meson-test.build. Similar to how we have "PGXS" build > support in the makefiles. Otherwise, I'm afraid this will get > annoying and error-prone if one has to remember to update other files > under src/test/ when adding for example a new .sql file to amcheck. I don't think we can use something other than meson.build. I solved that by editing the test_meson_extension script, now meson-test.build files live under the actual contrib/${extension}/ directory and the test script moves them to the correct directory. I needed to use the get_option('meson_source_dir') hack to get paths of the source files. > Also, the driver script is at 'src/tools/ci/test_meson_extensions', > but you are using it outside of CI, so that's not a good location. You are right, I moved the test_meson_extensions script under the 'src/tools/'. > v9-0003-meson-WIP-Add-docs-for-postgresql-extension.pc.patch > > Let's not rename existing ids. > > It seems to me that the .pc file can also be used without meson. > Let's take that into account a bit. For example, the > id="extend-postgres-meson" could be id="extend-postgres-pkg-config" or > similar. Sorry but I didn't understand how we can add a pkg-config documentation without renaming existing ids. 'Extension Building Infrastructure' is covered by <sect1 id="extend-pgxs">. I guess we would want to add pkg-config documentation under the extension building infrastructure, but it is something other than PGXS. So, it being under '<sect1 id="extend-pgxs">' doesn't sound correct to me. > Your text ends with a colon. Did you mean to add more text? Maybe an > example meson.build would be good. Yes, sorry for that. I added an example meson.build file. > v9-0005-meson-Add-LLVM-bitcode-emissions-for-contrib-libr.patch > > +# some libraries include "hstore/hstore.h" instead of "hstore.h" > > It seems to me that the former is correct, but if not then we should > fix it. I think both of them are correct and the comment is wrong. Source files in the contrib/hstore directory include "hstore.h" and files outside of this directory include "hstore/hstore.h". I changed this comment to '# Files outside of the current directory include hstore as "hstore/hstore.h"'. [1] https://mesonbuild.com/Pkgconfig-module.html#implicit-dependencies -- Regards, Nazir Bilal Yavuz Microsoft -
Re: meson vs. llvm bitcode files
Zsolt Parragi <zsolt.parragi@percona.com> — 2026-03-18T22:23:35Z
Hello! + bitcode_targets += custom_target( + targetname, + depends: [bitcode_obj], + input: [srcfile], + output: targetname, + command: [llvm_irgen_command, bitcode_cflags_gen_local], + install: true, + install_dir: dir_bitcode, + ) This seems overeager to rebuild to me. If I touch a single header file, it rebuilds all bc files because bitcode_obj (postgres_lib) changes. Wouldn't something like the following work? bitcode_targets += custom_target( targetname, depends: [generated_backend_headers_stamp], # instead of [postgres_lib] input: [srcfile], output: targetname, command: [llvm_irgen_command, llvm_irgen_dep_args, bitcode_cflags_local], # added llvm_irgen_dep_args depfile: targetname + '.d', # added install: true, install_dir: dir_bitcode, ) It seems to work in my testing, and results in less rebuilds. But I also have a fairly recent meson version, there might be some issues/limitations with earlier versions? +bitcode_modules += { + 'target': hstore_plpython, + 'srcfiles': hstore_plpython_sources, + 'additional_flags': [ + '-I@0@'.format(hstore_dir_up), + '-DPLPYTHON_LIBNAME="plpython3"', + '-I@0@'.format(python3_incdir), + '-I@0@'.format(plpython_dir), + perl_ccflags, + ] +} Do we need perl_ccflags for python? +meson_args = ' '.join(args.meson_args) ... + if meson_args: + meson_setup_command = [meson_bin, meson_args, 'setup', test_args, test_out_dir] + else: Will this properly work with multiple args? + +project('auth_delay', 'c') + Seems like a copy paste mistake, this is in postgres_fdw + if meson_args: + meson_setup_command = [meson_bin, meson_args, 'setup', test_args, test_out_dir] + else: + meson_setup_command = [meson_bin, 'setup', test_args, test_out_dir] + + meson_compile_command = ['meson', 'compile', '-C', test_out_dir, '-v'] last one should also be meson_bin + +exit_code = 0 + This is an unused variable src/makefiles/meson.build contains the following, this should be updated with this patch? # TODO: requires bitcode generation to be implemented for meson 'BITCODE_CFLAGS': '', 'BITCODE_CXXFLAGS': '', -
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-03-23T13:39:28Z
Hi! Thank you for looking into this! On Thu, 19 Mar 2026 at 01:23, Zsolt Parragi <zsolt.parragi@percona.com> wrote: > + bitcode_targets += custom_target( > + targetname, > + depends: [bitcode_obj], > + input: [srcfile], > + output: targetname, > + command: [llvm_irgen_command, bitcode_cflags_gen_local], > + install: true, > + install_dir: dir_bitcode, > + ) > > This seems overeager to rebuild to me. If I touch a single header > file, it rebuilds all bc files because bitcode_obj (postgres_lib) > changes. bitcode_obj is equal to postgres_lib only for postgres.index.bc. > Wouldn't something like the following work? > > bitcode_targets += custom_target( > targetname, > depends: [generated_backend_headers_stamp], # instead of [postgres_lib] > input: [srcfile], > output: targetname, > command: [llvm_irgen_command, llvm_irgen_dep_args, > bitcode_cflags_local], # added llvm_irgen_dep_args Could you mean 'bitcode_cflags_gen_local' instead of 'bitcode_cflags_local' here? Otherwise code doesn't compile. > depfile: targetname + '.d', # added > install: true, > install_dir: dir_bitcode, > ) > > It seems to work in my testing, and results in less rebuilds. But I > also have a fairly recent meson version, there might be some > issues/limitations with earlier versions? I couldn't reproduce this. There are 43 .bc files and header change (I changed simd.h) triggers a recompilation of only 'postgres.index.bc' on my end. This behavior seems correct to me. Could you please share steps to reproduce the behavior you saw? > +bitcode_modules += { > + 'target': hstore_plpython, > + 'srcfiles': hstore_plpython_sources, > + 'additional_flags': [ > + '-I@0@'.format(hstore_dir_up), > + '-DPLPYTHON_LIBNAME="plpython3"', > + '-I@0@'.format(python3_incdir), > + '-I@0@'.format(plpython_dir), > + perl_ccflags, > + ] > +} > > Do we need perl_ccflags for python? No, it seems I mixed plpython and plperl. Fixed. > +meson_args = ' '.join(args.meson_args) > ... > + if meson_args: > + meson_setup_command = [meson_bin, meson_args, 'setup', > test_args, test_out_dir] > + else: > > Will this properly work with multiple args? Updated. Previous version was merging all args into one string, now each argument is one string. > + > +project('auth_delay', 'c') > + > > Seems like a copy paste mistake, this is in postgres_fdw Fixed. > + if meson_args: > + meson_setup_command = [meson_bin, meson_args, 'setup', > test_args, test_out_dir] > + else: > + meson_setup_command = [meson_bin, 'setup', test_args, test_out_dir] > + > + meson_compile_command = ['meson', 'compile', '-C', test_out_dir, '-v'] > > last one should also be meson_bin Fixed. > + > +exit_code = 0 > + > > This is an unused variable Removed. > src/makefiles/meson.build contains the following, this should be > updated with this patch? > > # TODO: requires bitcode generation to be implemented for meson > 'BITCODE_CFLAGS': '', > 'BITCODE_CXXFLAGS': '', You are right, done. -- Regards, Nazir Bilal Yavuz Microsoft -
Re: meson vs. llvm bitcode files
Zsolt Parragi <zsolt.parragi@percona.com> — 2026-03-25T21:56:00Z
> I couldn't reproduce this. There are 43 .bc files and header change (I > changed simd.h) triggers a recompilation of only 'postgres.index.bc' > on my end. This behavior seems correct to me. Could you please share > steps to reproduce the behavior you saw? I can reliably reproduce it on multiple computers. patch applied on top of 009833ef2002430b1f8fae3a8e47718642b7a5d9, default meson build with only -Dllvm=enabled both on ubuntu 24.04/openSUSE tumbleweed with meson 1.3.2, ninja 1.11.1/1.13.2 ninja # to ensure I built everything ninja # nothing happens, up to date touch ../src/include/port/simd.h && time ninja -j$(nproc) 2>&1 [884/884] Generating src/backend/jit/llvm/bitcode/postgres with a custom command ninja -j$(nproc) 2>&1 188.04s user 31.36s system 4541% cpu 4.831 total touch ../src/include/port/simd.h && ninja -j$(nproc) 2>&1 -n | grep bc | wc -l 796
-
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-03-26T11:27:15Z
Hi, On Thu, 26 Mar 2026 at 00:56, Zsolt Parragi <zsolt.parragi@percona.com> wrote: > > > I couldn't reproduce this. There are 43 .bc files and header change (I > > changed simd.h) triggers a recompilation of only 'postgres.index.bc' > > on my end. This behavior seems correct to me. Could you please share > > steps to reproduce the behavior you saw? > > I can reliably reproduce it on multiple computers. > > patch applied on top of 009833ef2002430b1f8fae3a8e47718642b7a5d9, > default meson build with only -Dllvm=enabled > > both on ubuntu 24.04/openSUSE tumbleweed with meson 1.3.2, ninja 1.11.1/1.13.2 > > ninja # to ensure I built everything > ninja # nothing happens, up to date > > touch ../src/include/port/simd.h && time ninja -j$(nproc) 2>&1 > [884/884] Generating src/backend/jit/llvm/bitcode/postgres with a custom command > ninja -j$(nproc) 2>&1 188.04s user 31.36s system 4541% cpu 4.831 total > > touch ../src/include/port/simd.h && ninja -j$(nproc) 2>&1 -n | grep bc | wc -l > 796 Thank you! I am able to reproduce the problem and your suggested fix [1] fixes the problem for non-generated source files. However, when we use [1] for the generated sources, then some of the generated sources are added to the dependency (targetname + '.d') file. The problem is that generated sources in the dependency file do not exist in the source directory, they exist in the build directory as they are generated. For example: ``` $ninja -t missingdeps Missing dep: src/backend/jit/llvm/bitcode/postgres_src_backend_bootstrap_bootscanner_c.bc uses src/backend/bootstrap/bootparse.h (generated by CUSTOM_COMMAND) Missing dep: src/backend/jit/llvm/bitcode/postgres_src_backend_replication_repl_scanner_c.bc uses src/backend/replication/repl_gram.h (generated by CUSTOM_COMMAND) Missing dep: src/backend/jit/llvm/bitcode/postgres_src_backend_replication_syncrep_scanner_c.bc uses src/backend/replication/syncrep_gram.h (generated by CUSTOM_COMMAND) Missing dep: src/backend/jit/llvm/bitcode/cube_contrib_cube_cubescan_c.bc uses contrib/cube/cubeparse.h (generated by CUSTOM_COMMAND) Missing dep: src/backend/jit/llvm/bitcode/seg_contrib_seg_segscan_c.bc uses contrib/seg/segparse.h (generated by CUSTOM_COMMAND) Processed 3443 nodes. Error: There are 5 missing dependency paths. 5 targets had depfile dependencies on 5 distinct generated inputs (from 1 rules) without a non-depfile dep path to the generator. There might be build flakiness if any of the targets listed above are built alone, or not late enough, in a clean output directory. ``` cubeparse.h is doesn't exist in the source directory as default but it is generated by the: ``` cube_parse = custom_target('cubeparse', input: 'cubeparse.y', kwargs: bison_kw, ) ``` I solved that by adding them as extra dependencies: ``` bc_cube_gen_sources = [ { 'srcfiles': [cube_scan], 'extra_depends': [cube_parse] } ] ``` [1] bitcode_targets += custom_target( targetname, depends: [generated_backend_headers_stamp], # instead of [postgres_lib] input: [srcfile], output: targetname, command: [llvm_irgen_command, llvm_irgen_dep_args, bitcode_cflags_local], # added llvm_irgen_dep_args depfile: targetname + '.d', # added install: true, install_dir: dir_bitcode, ) -- Regards, Nazir Bilal Yavuz Microsoft -
Re: meson vs. llvm bitcode files
Peter Eisentraut <peter@eisentraut.org> — 2026-03-31T07:43:11Z
On 18.03.26 13:44, Nazir Bilal Yavuz wrote: >> The comment that introduces postgresql-extension-warnings.pc says >> >> +# Extension modules should likely also use -fwrapv etc. But it it's a >> bit odd >> +# to expose it to a .pc file? >> >> but then -fwrapv ends up in postgresql-extension.pc anyway. Not sure >> what was intended here. > > I asked Andres off-list and Andres said that we need to have these > flags inside the .pc file but it is not very nice since these flags > (-fwrapv for example) change the behavior. Maybe Andres could clarify > this better. Yes, it's probably right that extensions should build with the -f options that the server uses. You probably need -fwrapv and -fno-strict-aliasing at least. Then again, we don't know which compiler will consume the .pc file and whether it even supports these options in that particular spelling. >> The Requires list in my case is for example >> >> Requires: krb5-gssapi, icu-uc, icu-i18n, ldap, libxml-2.0 >= 2.6.23, >> liblz4, openssl, zlib, libzstd >= 1.4.0 >> >> but I don't think these are actually required for building extensions >> (unless a particular extension directly makes use of one of them, in >> which case they should declare that on their own). > > It seems that is how meson pkgconfig.generate() handles the > dependencies, please see [1]: > > ... > * Dependencies provided by pkg-config are added into Requires: or > Requires.private:. If a version was specified when declaring that > dependency it will be written into the generated file too. > ... Sure, but that doesn't make it right. ;-) It would be a quite a regression if extensions switched to using this .pc file (which we would want them to eventually), and then building an extension would require installing all these -dev packages. >> If we are going to install these .pc files, we also need to build them >> with with makefiles. Alternatively, we could not install them for now >> and just use them internally. > > Unfortunately, these .pc files are always installed in meson build. I > added a WIP patch (0007) for building .pc files with makefiles, I am > not sure if I am following the correct way. I would appreciate any > help on this. Seems reasonable as a start. (But there you have an empty Requires list.) >> I don't know if it's possible to make meson use a different file than >> meson.build, but if so, it might be better to keep these test >> meson.build files together with their extensions, like >> contrib/amcheck/meson-test.build. Similar to how we have "PGXS" build >> support in the makefiles. Otherwise, I'm afraid this will get >> annoying and error-prone if one has to remember to update other files >> under src/test/ when adding for example a new .sql file to amcheck. > > I don't think we can use something other than meson.build. I solved > that by editing the test_meson_extension script, now meson-test.build > files live under the actual contrib/${extension}/ directory and the > test script moves them to the correct directory. I needed to use the > get_option('meson_source_dir') hack to get paths of the source files. That seems more manageable. But I wonder whether it wouldn't be simpler to create a bespoke test module for testing this, rather than overlaying this onto production modules. That way, in the future, it would be easier to add more tests and variants and play around with this more freely without having to interfere with the real modules. >> v9-0003-meson-WIP-Add-docs-for-postgresql-extension.pc.patch >> >> Let's not rename existing ids. >> >> It seems to me that the .pc file can also be used without meson. >> Let's take that into account a bit. For example, the >> id="extend-postgres-meson" could be id="extend-postgres-pkg-config" or >> similar. > > Sorry but I didn't understand how we can add a pkg-config > documentation without renaming existing ids. 'Extension Building > Infrastructure' is covered by <sect1 id="extend-pgxs">. I guess we > would want to add pkg-config documentation under the extension > building infrastructure, but it is something other than PGXS. So, it > being under '<sect1 id="extend-pgxs">' doesn't sound correct to me. I mean stuff like - <varlistentry id="extend-pgxs-modules"> + <varlistentry id="extend-postgres-pgxs-modules"> seems unnecessary. You can keep the id of the existing section as "extend-pgxs" and add a new one in parallel. We don't need to have a perfect hierarchy of ids, especially if it means changing existing ones. I think at this point it's clear that this patch set needs quite a bit more consideration, so let's move it to PG20. -
Re: meson vs. llvm bitcode files
Andres Freund <andres@anarazel.de> — 2026-04-01T13:19:34Z
Hi, On 2026-03-31 09:43:11 +0200, Peter Eisentraut wrote: > On 18.03.26 13:44, Nazir Bilal Yavuz wrote: > > > The comment that introduces postgresql-extension-warnings.pc says > > > > > > +# Extension modules should likely also use -fwrapv etc. But it it's a > > > bit odd > > > +# to expose it to a .pc file? > > > > > > but then -fwrapv ends up in postgresql-extension.pc anyway. Not sure > > > what was intended here. > > > > I asked Andres off-list and Andres said that we need to have these > > flags inside the .pc file but it is not very nice since these flags > > (-fwrapv for example) change the behavior. Maybe Andres could clarify > > this better. > > Yes, it's probably right that extensions should build with the -f options > that the server uses. You probably need -fwrapv and -fno-strict-aliasing at > least. Then again, we don't know which compiler will consume the .pc file > and whether it even supports these options in that particular spelling. At the moment you can't really build server extensions against a postgres built with a different compiler. E.g. building extensions with clang against a server built with gcc triggers hundreds of warnings. We just do too many detections at configure time to make that reliably work. I don't think this is a meson specific problem... > > > The Requires list in my case is for example > > > > > > Requires: krb5-gssapi, icu-uc, icu-i18n, ldap, libxml-2.0 >= 2.6.23, > > > liblz4, openssl, zlib, libzstd >= 1.4.0 > > > > > > but I don't think these are actually required for building extensions > > > (unless a particular extension directly makes use of one of them, in > > > which case they should declare that on their own). > > > > It seems that is how meson pkgconfig.generate() handles the > > dependencies, please see [1]: > > > > ... > > * Dependencies provided by pkg-config are added into Requires: or > > Requires.private:. If a version was specified when declaring that > > dependency it will be written into the generated file too. > > ... > > Sure, but that doesn't make it right. ;-) It would be a quite a regression > if extensions switched to using this .pc file (which we would want them to > eventually), and then building an extension would require installing all > these -dev packages. I'm pretty confused - isn't what meson does the only reasonable thing? You can't reliably build an extension without having all the dependencies that the server was built against also installed, because we have headers that include the dependencies. If you e.g. don't have openssl or kerberos installed any [indirect ]include of libpq-be.h will fail. I don't understand how this would be a regression. And adding these dependencies to eitehr Requires or Requires.private seems pretty much required to ever get static linking to work. Greetings, Andres Freund
-
Re: meson vs. llvm bitcode files
Andres Freund <andres@anarazel.de> — 2026-04-06T20:09:26Z
Hi, On 2026-03-26 14:27:15 +0300, Nazir Bilal Yavuz wrote: > From ca280ffa86c4b4804ce517414c7aae015d6d21ea Mon Sep 17 00:00:00 2001 > From: Andres Freund <andres@anarazel.de> > Date: Sat, 27 Aug 2022 09:52:03 -0700 > Subject: [PATCH v12 1/7] meson: Add postgresql-extension.pc for building > extension libraries I don't think we have enough agreement for this with a name as general as this yet. What if we just have a postgresql-llvm-jit-bitcode or postgresql-$version-llvm-jit-bitcode for now? Then we also don't need the tests and the autoconf parts yet. > Subject: [PATCH v12 4/7] meson: Add architecture for LLVM bitcode emission > +foreach bitcode_module : bitcode_modules > + bitcode_targets = [] > + bitcode_obj = bitcode_module['target'] > + bitcode_cflags_local = bitcode_cflags + bitcode_module.get('additional_flags', []) > + bitcode_name = bitcode_module.get('name', bitcode_obj.name()) > + > + foreach srcfile : bitcode_module['srcfiles'] > + if meson.version().version_compare('>=0.59') > + srcfilename = fs.parent(srcfile) / fs.name(srcfile) > + else > + srcfilename = '@0@'.format(srcfile) > + endif > + > + targetname = '@0@_@1@.bc'.format( > + bitcode_name, > + srcfilename.underscorify(), > + ) > + bitcode_targets += custom_target( > + targetname, > + depends: [generated_backend_headers_stamp], One thing that's quite annoying about the dependencies is that somehow this makes meson a lot slower generating build.ninja. It takes 22s for me with the depend present, 5.1s without. One thing that seems to be fast, but is also somewhat ugly, is to use no depends but input: [srcfile, bitcode_obj.extract_objects(srcfile)], That works because llvm_irgen_command just uses '@INPUT0@', but still adds a dependency to the .o file, which in turn has the right dependencies. While a bit gross, it still seems far better than 4x ing the build.ninja generation time. > + # Process generated sources, which may include custom compilation flags. > + foreach gen_sources: bitcode_module.get('gen_sources', []) Are there actually cases where it makes sense to generate bitcode for these? I'm a bit doubtful. > From a4559525c8e9fa2999bf1d151dfe17bb83dc50f7 Mon Sep 17 00:00:00 2001 > From: Nazir Bilal Yavuz <byavuz81@gmail.com> > Date: Mon, 16 Mar 2026 18:15:59 +0300 > Subject: [PATCH v12 5/7] meson: Add LLVM bitcode emissions for contrib > libraries > > The libraries which the bitcode files will be generated in are selected > manually. > > Author: Andres Freund <andres@anarazel.de> > Author: Nazir Bilal Yavuz <byavuz81@gmail.com> > Author: Diego Fronza <diego.fronza@percona.com> > Reviewed-by: Peter Eisentraut <peter@eisentraut.org> > Reviewed-by: Diego Fronza <diego.fronza@percona.com> > Reviewed-by: Zsolt Parragi <zsolt.parragi@percona.com> > Discussion: https://postgr.es/m/206b001d-1884-4081-bd02-bed5c92f02ba%40eisentraut.org > --- > contrib/bloom/meson.build | 5 +++++ > contrib/bool_plperl/meson.build | 9 +++++++++ > contrib/btree_gin/meson.build | 5 +++++ > contrib/btree_gist/meson.build | 5 +++++ > contrib/citext/meson.build | 5 +++++ > contrib/cube/meson.build | 18 ++++++++++++++++++ > contrib/dict_int/meson.build | 5 +++++ > contrib/dict_xsyn/meson.build | 5 +++++ > contrib/earthdistance/meson.build | 6 ++++++ > contrib/fuzzystrmatch/meson.build | 9 +++++++++ > contrib/hstore/meson.build | 7 +++++++ > contrib/hstore_plperl/meson.build | 10 ++++++++++ > contrib/hstore_plpython/meson.build | 11 +++++++++++ > contrib/intarray/meson.build | 5 +++++ > contrib/isn/meson.build | 5 +++++ > contrib/jsonb_plperl/meson.build | 8 ++++++++ > contrib/jsonb_plpython/meson.build | 10 ++++++++++ > contrib/lo/meson.build | 5 +++++ > contrib/ltree/meson.build | 10 ++++++++++ > contrib/ltree_plpython/meson.build | 11 +++++++++++ > contrib/pg_buffercache/meson.build | 5 +++++ > contrib/pg_freespacemap/meson.build | 5 +++++ > contrib/pg_logicalinspect/meson.build | 5 +++++ > contrib/pg_surgery/meson.build | 4 ++++ > contrib/pg_trgm/meson.build | 5 +++++ > contrib/pgcrypto/meson.build | 4 ++++ > contrib/seg/meson.build | 12 ++++++++++++ > contrib/spi/meson.build | 20 ++++++++++++++++++++ > contrib/sslinfo/meson.build | 5 +++++ > contrib/tablefunc/meson.build | 5 +++++ > contrib/tcn/meson.build | 5 +++++ > contrib/tsm_system_rows/meson.build | 5 +++++ > contrib/tsm_system_time/meson.build | 5 +++++ > contrib/unaccent/meson.build | 5 +++++ > contrib/uuid-ossp/meson.build | 5 +++++ > contrib/xml2/meson.build | 5 +++++ > meson.build | 1 + > src/interfaces/libpq/meson.build | 3 +++ > src/pl/plperl/meson.build | 1 + > src/pl/plpython/meson.build | 1 + > 40 files changed, 260 insertions(+) I think for most of these don't make much sense to generate bitcode. I'd probably restrict it to hstore, citext, intarray, ltree, pg_trgm or such. There need to have lightweight SQL operators / functions to be considered for inlining, and that's just not the usecase most of these have. > From aa64b9ffc2b6aed9057568d616ab9e6c43af4b16 Mon Sep 17 00:00:00 2001 > From: Nazir Bilal Yavuz <byavuz81@gmail.com> > Date: Wed, 12 Mar 2025 10:44:46 +0300 > Subject: [PATCH v12 6/7] meson: Add LLVM bitcode emission for backend sources > > Since generated backend sources may have their own compilation flags and > must also be included in the postgres.index.bc, the way to make it work > with current code was to create a new variable, called > `bc_generated_backend_sources`, which is a list of dictionaries, each > one having an optional 'additional_flags' and a `srclist` pointing to > the list of custom_target generated sources. > > An example of a possible structure of bitcode_modules which is processed > by the main meson llvm bitcode emission file > src/backend/jit/llvm/bitcode/meson.build: I'm a bit doubtful this is worth doing, I don't think anything in there could be potentially inlineable... Greetings, Andres Freund -
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-07-01T10:25:00Z
Hi, Thanks for looking into this! On Mon, 6 Apr 2026 at 23:09, Andres Freund <andres@anarazel.de> wrote: > > On 2026-03-26 14:27:15 +0300, Nazir Bilal Yavuz wrote: > > From ca280ffa86c4b4804ce517414c7aae015d6d21ea Mon Sep 17 00:00:00 2001 > > From: Andres Freund <andres@anarazel.de> > > Date: Sat, 27 Aug 2022 09:52:03 -0700 > > Subject: [PATCH v12 1/7] meson: Add postgresql-extension.pc for building > > extension libraries > > I don't think we have enough agreement for this with a name as general as this > yet. What if we just have a postgresql-llvm-jit-bitcode or > postgresql-$version-llvm-jit-bitcode for now? Renamed to 'postgresql-$version-llvm-jit-bitcode'. > Then we also don't need the tests and the autoconf parts yet. These patches are removed. > > Subject: [PATCH v12 4/7] meson: Add architecture for LLVM bitcode emission > > > +foreach bitcode_module : bitcode_modules > > + bitcode_targets = [] > > + bitcode_obj = bitcode_module['target'] > > + bitcode_cflags_local = bitcode_cflags + bitcode_module.get('additional_flags', []) > > + bitcode_name = bitcode_module.get('name', bitcode_obj.name()) > > + > > + foreach srcfile : bitcode_module['srcfiles'] > > + if meson.version().version_compare('>=0.59') > > + srcfilename = fs.parent(srcfile) / fs.name(srcfile) > > + else > > + srcfilename = '@0@'.format(srcfile) > > + endif > > + > > + targetname = '@0@_@1@.bc'.format( > > + bitcode_name, > > + srcfilename.underscorify(), > > + ) > > + bitcode_targets += custom_target( > > + targetname, > > + depends: [generated_backend_headers_stamp], > > One thing that's quite annoying about the dependencies is that somehow this > makes meson a lot slower generating build.ninja. It takes 22s for me with the > depend present, 5.1s without. > > One thing that seems to be fast, but is also somewhat ugly, is to use no > depends but > input: [srcfile, bitcode_obj.extract_objects(srcfile)], > > That works because llvm_irgen_command just uses '@INPUT0@', but still adds a > dependency to the .o file, which in turn has the right dependencies. > > While a bit gross, it still seems far better than 4x ing the build.ninja > generation time. I am able to reproduce it, done. > > + # Process generated sources, which may include custom compilation flags. > > + foreach gen_sources: bitcode_module.get('gen_sources', []) > > Are there actually cases where it makes sense to generate bitcode for these? > I'm a bit doubtful. Removed, this is not needed for the libraries you mentioned below. > > From a4559525c8e9fa2999bf1d151dfe17bb83dc50f7 Mon Sep 17 00:00:00 2001 > > From: Nazir Bilal Yavuz <byavuz81@gmail.com> > > Date: Mon, 16 Mar 2026 18:15:59 +0300 > > Subject: [PATCH v12 5/7] meson: Add LLVM bitcode emissions for contrib > > libraries > > > > The libraries which the bitcode files will be generated in are selected > > manually. > > > > Author: Andres Freund <andres@anarazel.de> > > Author: Nazir Bilal Yavuz <byavuz81@gmail.com> > > Author: Diego Fronza <diego.fronza@percona.com> > > Reviewed-by: Peter Eisentraut <peter@eisentraut.org> > > Reviewed-by: Diego Fronza <diego.fronza@percona.com> > > Reviewed-by: Zsolt Parragi <zsolt.parragi@percona.com> > > Discussion: https://postgr.es/m/206b001d-1884-4081-bd02-bed5c92f02ba%40eisentraut.org > > --- ... > > I think for most of these don't make much sense to generate bitcode. I'd > probably restrict it to hstore, citext, intarray, ltree, pg_trgm or > such. There need to have lightweight SQL operators / functions to be > considered for inlining, and that's just not the usecase most of these have. Done, only 'hstore, citext, intarray, ltree and pg_trgm' are inlined for now. > > From aa64b9ffc2b6aed9057568d616ab9e6c43af4b16 Mon Sep 17 00:00:00 2001 > > From: Nazir Bilal Yavuz <byavuz81@gmail.com> > > Date: Wed, 12 Mar 2025 10:44:46 +0300 > > Subject: [PATCH v12 6/7] meson: Add LLVM bitcode emission for backend sources > > > > Since generated backend sources may have their own compilation flags and > > must also be included in the postgres.index.bc, the way to make it work > > with current code was to create a new variable, called > > `bc_generated_backend_sources`, which is a list of dictionaries, each > > one having an optional 'additional_flags' and a `srclist` pointing to > > the list of custom_target generated sources. > > > > An example of a possible structure of bitcode_modules which is processed > > by the main meson llvm bitcode emission file > > src/backend/jit/llvm/bitcode/meson.build: > > I'm a bit doubtful this is worth doing, I don't think anything in there could > be potentially inlineable... Removed. v13 is attached, it is much smaller compared to previous versions. I liked that because I think it is easy to work on this feature now, we can incrementally expand the feature later. -- Regards, Nazir Bilal Yavuz Microsoft -
Re: meson vs. llvm bitcode files
Zsolt Parragi <zsolt.parragi@percona.com> — 2026-07-02T21:50:13Z
Everything seems to be properly working with the new version in my tests, I only have a few minor comments: + <para> + When <productname>PostgreSQL</productname> is built with + <productname>Meson</productname>, it installs a + <literal>pkg-config</literal> file named + <literal>postgresql-&majorversion;-llvm-jit-bitcode.pc</literal>. It is + used internally to emit <acronym>LLVM</acronym> <acronym>JIT</acronym> + bitcode for the server and for selected + <filename>contrib</filename> modules, and it collects the same + compilation flags that are used to build the server itself. + </para> Do we want to install these pc files even with -Dllvm=disabled? diff --git a/src/tools/irlink b/src/tools/irlink new file mode 100644 index 00000000000..6a03e4f5695 --- /dev/null +++ b/src/tools/irlink Shouldn't this file committed as executable? +import shutil +import subprocess +import sys Is shutil and sys used in the script?
-
Re: meson vs. llvm bitcode files
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-07-06T13:37:40Z
Hi, Sorry, I didn't notice your reply as it is sent to -hackers list only. Thank you for looking into this! On Fri, 3 Jul 2026 at 00:50, Zsolt Parragi <zsolt.parragi@percona.com> wrote: > > Everything seems to be properly working with the new version in my > tests, I only have a few minor comments: > > + <para> > + When <productname>PostgreSQL</productname> is built with > + <productname>Meson</productname>, it installs a > + <literal>pkg-config</literal> file named > + <literal>postgresql-&majorversion;-llvm-jit-bitcode.pc</literal>. It is > + used internally to emit <acronym>LLVM</acronym> <acronym>JIT</acronym> > + bitcode for the server and for selected > + <filename>contrib</filename> modules, and it collects the same > + compilation flags that are used to build the server itself. > + </para> > > Do we want to install these pc files even with -Dllvm=disabled? I think we want to install them as these pc files can be used for building extensions. > diff --git a/src/tools/irlink b/src/tools/irlink > new file mode 100644 > index 00000000000..6a03e4f5695 > --- /dev/null > +++ b/src/tools/irlink > > Shouldn't this file committed as executable? Yes, done. > +import shutil > +import subprocess > +import sys > > Is shutil and sys used in the script? They are not used, removed. v14 is attached. -- Regards, Nazir Bilal Yavuz Microsoft