Thread

1. List TAP test files in makefiles

   Peter Eisentraut <peter@eisentraut.org> — 2025-08-23T06:09:50Z

   meson.build files have to list TAP test files explicitly.  Makefiles 
   have been relying on a t/*.pl wildcard.  As a consequence, it is 
   traditional now that many developers who are primarily using make forget 
   to add any new TAP test files to the respective meson.build file.
   
   The obvious solution is to also require the makefiles to list TAP files 
   explicitly, which is what I'm proposing here.
   
   To list the test files, I'm re-using the make variable TAP_TESTS that is 
   already in use by extensions.  Extensions up to now would just write 
   something like
   
        TAP_TESTS = 1
   
   Starting with this patch, they would have to write
   
        TAP_TESTS = t/001_foo.pl t/002_bar.pl
   
   This makes it somewhat backward compatible.  (For example, with the 
   latter formulation, PG<=18 would still read it as enabling TAP tests, 
   but it would ignore the actual file list.)
   
   (I have considered other variants: We already have the make variable 
   PROVE_TESTS that you can use to override the list of tests from the make 
   command line.  But if you use that one to list the tests in an extension 
   makefile, then that extension would have to use both PROVE_TESTS and 
   TAP_TESTS (to enable TAP tests overall), which seems pretty confusing. 
   I think the solution I ended up with is compact and intuitive.)
   
   In the attached patch, I have arranged it so that the interesting 
   changes are in the first three files, the rest is just mechanical.

2. Re: List TAP test files in makefiles

   x4mmm@yandex-team.ru — 2025-08-23T06:57:37Z

   I agree that this difference between meson and autotools builds is kind of problematic.
   
   > On 23 Aug 2025, at 11:09, Peter Eisentraut <peter@eisentraut.org> wrote:
   > 
   > The obvious solution is to also require the makefiles to list TAP files explicitly, which is what I'm proposing here.
   
   What is the downside of the approach where meson uses t/*.pl wildcard?
   
   
   Best regards, Andrey Borodin.
   
   
   

3. Re: List TAP test files in makefiles

   Andres Freund <andres@anarazel.de> — 2025-08-23T14:02:32Z

   Hi,
   
   On 2025-08-23 11:57:37 +0500, Andrey Borodin wrote:
   > I agree that this difference between meson and autotools builds is kind of problematic.
   > 
   > > On 23 Aug 2025, at 11:09, Peter Eisentraut <peter@eisentraut.org> wrote:
   > > 
   > > The obvious solution is to also require the makefiles to list TAP files explicitly, which is what I'm proposing here.
   > 
   > What is the downside of the approach where meson uses t/*.pl wildcard?
   
   In meson you can't do wildcards at "configure" time, since wildcards do not
   allow reliable detection of when re-configure is needed.  You could do it by
   running prove to run the tap tests, like make does, but that would
   considerably slow down the tests, as prove has either no parallelism or
   independent parallelism from the make's (or ninjas).
   
   Greetings,
   
   Andres Freund
   
   
   
   

4. Re: List TAP test files in makefiles

   Tom Lane <tgl@sss.pgh.pa.us> — 2025-08-23T14:17:59Z

   Andres Freund <andres@anarazel.de> writes:
   > On 2025-08-23 11:57:37 +0500, Andrey Borodin wrote:
   >> What is the downside of the approach where meson uses t/*.pl wildcard?
   
   > In meson you can't do wildcards at "configure" time, since wildcards do not
   > allow reliable detection of when re-configure is needed.
   
   But ... what "re-configuration" is needed when adding a new test file?
   
   This proposed fix adds more manual maintenance effort, and I really
   doubt that it will do much to solve the ostensible problem that
   people forget to update the meson.build files.  We should be striving
   to make the meson system as easy to use as autoconf/make, not trying
   to attain parity by making the latter experience worse.
   
   			regards, tom lane
   
   
   
   

5. Re: List TAP test files in makefiles

   Andres Freund <andres@anarazel.de> — 2025-08-23T14:38:09Z

   Hi,
   
   On 2025-08-23 10:17:59 -0400, Tom Lane wrote:
   > Andres Freund <andres@anarazel.de> writes:
   > > On 2025-08-23 11:57:37 +0500, Andrey Borodin wrote:
   > >> What is the downside of the approach where meson uses t/*.pl wildcard?
   > 
   > > In meson you can't do wildcards at "configure" time, since wildcards do not
   > > allow reliable detection of when re-configure is needed.
   > 
   > But ... what "re-configuration" is needed when adding a new test file?
   
   In meson, not autoconf/make. The test runner doesn't parse the meson.build
   files, that's done once when running configure. After that there's a table of
   tests that need to be run that gets used when you type ninja/meson test.
   
   
   > We should be striving to make the meson system as easy to use as
   > autoconf/make, not trying to attain parity by making the latter experience
   > worse.
   
   FWIW, I find the autoconf/make test run experience completely unusable. It
   literally is made me embark on getting away from it. I don't understand how
   people stand it.
   
   Greetings,
   
   Andres Freund
   
   
   
   

6. Re: List TAP test files in makefiles

   Andrew Dunstan <andrew@dunslane.net> — 2025-08-23T15:23:21Z

   On 2025-08-23 Sa 10:38 AM, Andres Freund wrote:
   > Hi,
   >
   > On 2025-08-23 10:17:59 -0400, Tom Lane wrote:
   >
   >> We should be striving to make the meson system as easy to use as
   >> autoconf/make, not trying to attain parity by making the latter experience
   >> worse.
   > FWIW, I find the autoconf/make test run experience completely unusable. It
   > literally is made me embark on getting away from it. I don't understand how
   > people stand it.
   >
   
   
   Honestly, they both have annoyances and points where it's easy to get 
   into the weeds. There are definitely cases where meson has left me 
   scratching my head.
   
   But I think this is really analogous to adding a new object file. That 
   needs to be added to the Makefile as well as the meson.build file, and I 
   don't see why tests shouldn't work the same. That might be slightly 
   annoying, but it would be way down on my list.
   
   
   cheers
   
   
   andrew
   
   
   --
   Andrew Dunstan
   EDB: https://www.enterprisedb.com
   
   
   
   
   

7. Re: List TAP test files in makefiles

   Tom Lane <tgl@sss.pgh.pa.us> — 2025-08-23T16:11:51Z

   Andres Freund <andres@anarazel.de> writes:
   > FWIW, I find the autoconf/make test run experience completely unusable. It
   > literally is made me embark on getting away from it. I don't understand how
   > people stand it.
   
   Interesting perspective, because from where I sit the testing aspect
   is the weakest part of our meson setup: it seems to be very much
   all-or-nothing, whereas the makefiles allow you to run any particular
   test suite you want.  Maybe this is just a lack of documentation?
   There is still not a word in
   
   https://www.postgresql.org/docs/devel/regress-run.html
   
   about how to accomplish any of those effects under meson.
   
   I find the lack of any equivalent to installcheck to be
   particularly distressing, since it means you can't repeat
   a test without paying for "make temp-install" every time.
   Admittedly that doesn't work for a test that needs particular
   non-default installation settings, but that's a minority
   of our tests.
   
   			regards, tom lane
   
   
   
   

8. Re: List TAP test files in makefiles

   Andres Freund <andres@anarazel.de> — 2025-08-25T21:10:12Z

   Hi,
   
   On 2025-08-23 12:11:51 -0400, Tom Lane wrote:
   > Andres Freund <andres@anarazel.de> writes:
   > > FWIW, I find the autoconf/make test run experience completely unusable. It
   > > literally is made me embark on getting away from it. I don't understand how
   > > people stand it.
   >
   > Interesting perspective, because from where I sit the testing aspect
   > is the weakest part of our meson setup: it seems to be very much
   > all-or-nothing, whereas the makefiles allow you to run any particular
   > test suite you want.
   
   Meson definitely allows to run just some of the test. You can list all tests
   with
   meson test --list
   
   And you can run testsuites (like all the recovery tests) with
     meson test --suite recovery
   or individual tests with
     meson test test_misc/003_check_guc
   
   
   > Maybe this is just a lack of documentation?
   
   Yea. Partially because that's generic information about meson (i.e. not
   postgres specific stuff) and partially because there were some doc patches
   that we ended up not finding agreement upon approach-wise.
   
   
   > I find the lack of any equivalent to installcheck to be
   > particularly distressing, since it means you can't repeat
   > a test without paying for "make temp-install" every time.
   
   If you just want to skip the install (reusing the prior install), you can do
   that with
    meson test --no-suite setup
   
   That's basically the equivalent of NO_TEMP_INSTALL (which isn't documented
   either :( )
   
   
   > Admittedly that doesn't work for a test that needs particular
   > non-default installation settings, but that's a minority
   > of our tests.
   
   You can run all tests against a running postgres by doing
     meson test --setup running
   
   
   Unless somebody else volunteers (please!), I guess I should write up a
   patch...
   
   Greetings,
   
   Andres Freund
   
   
   
   

9. Re: List TAP test files in makefiles

   Álvaro Herrera <alvherre@kurilemu.de> — 2025-08-27T16:07:59Z

   On 2025-Aug-25, Andres Freund wrote:
   
   > Unless somebody else volunteers (please!), I guess I should write up a
   > patch...
   
   Maybe it doesn't have to be a patch -- we have some info on command
   lines to use for testing at
   https://wiki.postgresql.org/wiki/Meson#Test_related_commands
   which can perhaps be updated and filled in with additional content.
   
   There may be other outdated things in that page, which perhaps should be
   updated.  For instance, I would rip out the entire "Why and What"
   section, which feels no longer relevant.
   
   I'm a bit short of time at present, but in a few weeks I can give this a
   look.  I'm happy if somebody else does it though.
   
   -- 
   Álvaro Herrera               48°01'N 7°57'E  —  https://www.EnterpriseDB.com/
   Si no sabes adonde vas, es muy probable que acabes en otra parte.
   
   
   
   

10. Re: List TAP test files in makefiles

    Peter Eisentraut <peter@eisentraut.org> — 2025-09-01T05:09:13Z

    On 23.08.25 08:09, Peter Eisentraut wrote:
    > meson.build files have to list TAP test files explicitly.  Makefiles 
    > have been relying on a t/*.pl wildcard.  As a consequence, it is 
    > traditional now that many developers who are primarily using make forget 
    > to add any new TAP test files to the respective meson.build file.
    > 
    > The obvious solution is to also require the makefiles to list TAP files 
    > explicitly, which is what I'm proposing here.
    > 
    > To list the test files, I'm re-using the make variable TAP_TESTS that is 
    > already in use by extensions.  Extensions up to now would just write 
    > something like
    > 
    >      TAP_TESTS = 1
    > 
    > Starting with this patch, they would have to write
    > 
    >      TAP_TESTS = t/001_foo.pl t/002_bar.pl
    > 
    > This makes it somewhat backward compatible.  (For example, with the 
    > latter formulation, PG<=18 would still read it as enabling TAP tests, 
    > but it would ignore the actual file list.)
    > 
    > (I have considered other variants: We already have the make variable 
    > PROVE_TESTS that you can use to override the list of tests from the make 
    > command line.  But if you use that one to list the tests in an extension 
    > makefile, then that extension would have to use both PROVE_TESTS and 
    > TAP_TESTS (to enable TAP tests overall), which seems pretty confusing. I 
    > think the solution I ended up with is compact and intuitive.)
    > 
    > In the attached patch, I have arranged it so that the interesting 
    > changes are in the first three files, the rest is just mechanical.
    
    In the meantime, here is a new version to adapt to the addition of TAP 
    tests for libpq-oauth.

11. Re: List TAP test files in makefiles

    Nathan Bossart <nathandbossart@gmail.com> — 2025-10-24T15:49:53Z

    On Sat, Aug 23, 2025 at 10:17:59AM -0400, Tom Lane wrote:
    > This proposed fix adds more manual maintenance effort, and I really
    > doubt that it will do much to solve the ostensible problem that
    > people forget to update the meson.build files.
    
    This was my reaction, too.  I suspect folks will update either Makefile or
    meson.build based on what they use and then still forget to update the
    other one.
    
    -- 
    nathan
    
    
    
    

12. Re: List TAP test files in makefiles

    Michael Paquier <michael@paquier.xyz> — 2025-10-28T04:18:51Z

    On Fri, Oct 24, 2025 at 10:49:53AM -0500, Nathan Bossart wrote:
    > On Sat, Aug 23, 2025 at 10:17:59AM -0400, Tom Lane wrote:
    >> This proposed fix adds more manual maintenance effort, and I really
    >> doubt that it will do much to solve the ostensible problem that
    >> people forget to update the meson.build files.
    > 
    > This was my reaction, too.  I suspect folks will update either Makefile or
    > meson.build based on what they use and then still forget to update the
    > other one.
    
    This is not a problem limited to committers: it is a worse an issue
    for patch authors and reviewers.  I've seen it again literally three
    days ago, where a test was missing from meson, and the CI reported
    green without testing the patch in most environments where we use only
    meson, aka Windows.
    
    IMO, we should tackle this problem the other way around: what would it
    take to have meson build a list of the TAP files in t/ based on an
    automated rule when tests.tap is defined?
    
    Another less invasive solution for the code tree would be to have
    meson cross-check that the list of files it defines matches with
    what's on disk.  With the drop of Makefiles and ./configure at some
    point in mind, I am not convinced that the t/*.pl rule needs to
    change, but meson could be made more reliable by making sure that
    what's listed matches with  what we have in the source tree.  That
    would prevent a lot of mistakes.
    --
    Michael
    

13. Re: List TAP test files in makefiles

    Álvaro Herrera <alvherre@kurilemu.de> — 2025-10-31T16:15:14Z

    On 2025-Oct-28, Michael Paquier wrote:
    
    > IMO, we should tackle this problem the other way around: what would it
    > take to have meson build a list of the TAP files in t/ based on an
    > automated rule when tests.tap is defined?
    
    Maybe what we need is a single source of truth that is used both by
    Makefiles and meson: say, a file named "testlist" which contains lines
    like
    
    regress = foo bar baz
    tap = t/001_foo.pl t/002_bar.pl
    
    (possibly with backslash-escaped newlines) and so on, which is processed
    by each build system in the way that it needs.  This way, whichever
    system is used by a developer, either the test will run because they
    changed this one file, or it won't.
    
    I don't know how much scaffolding is needed to make this work, but I
    don't think it would be too bad.
    
    IIRC Andres has complained about the Makefiles using t/*.pl to determine
    the tests to run.  This would also solve that problem.
    
    -- 
    Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
    "I suspect most samba developers are already technically insane...
    Of course, since many of them are Australians, you can't tell." (L. Torvalds)
    
    
    
    

14. Re: List TAP test files in makefiles

    Nazir Bilal Yavuz <byavuz81@gmail.com> — 2025-11-02T20:35:44Z

    Hi,
    
    On Tue, 28 Oct 2025 at 07:19, Michael Paquier <michael@paquier.xyz> wrote:
    >
    > Another less invasive solution for the code tree would be to have
    > meson cross-check that the list of files it defines matches with
    > what's on disk.  With the drop of Makefiles and ./configure at some
    > point in mind, I am not convinced that the t/*.pl rule needs to
    > change, but meson could be made more reliable by making sure that
    > what's listed matches with  what we have in the source tree.  That
    > would prevent a lot of mistakes.
    
    I tried to implement this but it didn't quite work, please see details below.
    
    I introduced a new test, meson_tap_test_check, in the meson build
    system. This test runs first in the test suite and compares two file
    sets:
    
    1-  Files on disk matching the t/*.pl wildcard.
    2- The output of "meson introspect ${pg_root_dir} --tests" filtered
    with the same t/*.pl wildcard.
    
    The test subtracts the second list from the first and fails if there
    is any difference.
    
    However, this approach didn’t work because some tests are not
    registered if their dependencies (or such) aren’t found. For example,
    '001_sepgsql.pl' test isn't registered if the 'selinux' dependency is
    missing and that causes a difference between #1 and #2. Given that, I
    think it would be better if we register all tests and skip those whose
    dependencies aren’t found, rather than not registering them when the
    dependency isn’t found.
    
    The attached patch shows what I implemented.
    
    --
    Regards,
    Nazir Bilal Yavuz
    Microsoft
    

15. Re: List TAP test files in makefiles

    Peter Eisentraut <peter@eisentraut.org> — 2025-12-03T10:54:39Z

    On 01.09.25 07:09, Peter Eisentraut wrote:
    > On 23.08.25 08:09, Peter Eisentraut wrote:
    >> meson.build files have to list TAP test files explicitly.  Makefiles 
    >> have been relying on a t/*.pl wildcard.  As a consequence, it is 
    >> traditional now that many developers who are primarily using make 
    >> forget to add any new TAP test files to the respective meson.build file.
    >>
    >> The obvious solution is to also require the makefiles to list TAP 
    >> files explicitly, which is what I'm proposing here.
    >>
    >> To list the test files, I'm re-using the make variable TAP_TESTS that 
    >> is already in use by extensions.  Extensions up to now would just 
    >> write something like
    >>
    >>      TAP_TESTS = 1
    >>
    >> Starting with this patch, they would have to write
    >>
    >>      TAP_TESTS = t/001_foo.pl t/002_bar.pl
    >>
    >> This makes it somewhat backward compatible.  (For example, with the 
    >> latter formulation, PG<=18 would still read it as enabling TAP tests, 
    >> but it would ignore the actual file list.)
    >>
    >> (I have considered other variants: We already have the make variable 
    >> PROVE_TESTS that you can use to override the list of tests from the 
    >> make command line.  But if you use that one to list the tests in an 
    >> extension makefile, then that extension would have to use both 
    >> PROVE_TESTS and TAP_TESTS (to enable TAP tests overall), which seems 
    >> pretty confusing. I think the solution I ended up with is compact and 
    >> intuitive.)
    >>
    >> In the attached patch, I have arranged it so that the interesting 
    >> changes are in the first three files, the rest is just mechanical.
    > 
    > In the meantime, here is a new version to adapt to the addition of TAP 
    > tests for libpq-oauth.
    
    It looks like there is no consensus to go into this direction, and none 
    of the discussed alternatives are taking on a concrete shape, so I'm 
    withdrawing this patch.