Horton

Thread

Commits

  1. Explicitly skip TAP tests under Meson if disabled

  1. Explicitly skip TAP tests under Meson if disabled

    Peter Eisentraut <peter@eisentraut.org> — 2023-10-30T09:45:52Z

    Under Meson, it is not very easy to see if TAP tests have been enabled 
or disabled, if you rely on the default auto setting.  You either need 
to carefully study the meson setup output, or you notice, what a minute, 
didn't there use to be like 250 tests, not only 80?

I think it would be better if we still registered the TAP tests in Meson 
even if the tap_tests option is disabled, but with a dummy command that 
registers them as skipped.  That way you get a more informative output like

Ok:                 78
Expected Fail:      0
Fail:               0
Unexpected Pass:    0
Skipped:            187
Timeout:            0

which is really a more accurate representation of what the test run 
actually accomplished than "everything Ok".

See attached patch for a possible implementation.  (This uses perl as a 
hard build requirement.  We are planning to do that anyway, but 
obviously other implementations, such as using python, would also be 
possible.)

  2. Re: Explicitly skip TAP tests under Meson if disabled

    Aleksander Alekseev <aleksander@timescale.com> — 2023-10-30T13:47:14Z

    Hi,

> Under Meson, it is not very easy to see if TAP tests have been enabled
> or disabled, if you rely on the default auto setting.  You either need
> to carefully study the meson setup output, or you notice, what a minute,
> didn't there use to be like 250 tests, not only 80?
>
> I think it would be better if we still registered the TAP tests in Meson
> even if the tap_tests option is disabled, but with a dummy command that
> registers them as skipped.  That way you get a more informative output like
>
> Ok:                 78
> Expected Fail:      0
> Fail:               0
> Unexpected Pass:    0
> Skipped:            187
> Timeout:            0
>
> which is really a more accurate representation of what the test run
> actually accomplished than "everything Ok".
>
> See attached patch for a possible implementation.  (This uses perl as a
> hard build requirement.  We are planning to do that anyway, but
> obviously other implementations, such as using python, would also be
> possible.)

I tested the patch and it works as intended.

Personally I like the change. It makes the output more explicit. In my
use cases not running TAP tests typically is not something I want . So
I would appreciate being warned with a long list of bright yellow
"SKIP" messages. If I really want to skip TAP tests these messages are
just informative and don't bother me.

+1

-- 
Best regards,
Aleksander Alekseev

  3. Re: Explicitly skip TAP tests under Meson if disabled

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-10-30T14:12:46Z

    Aleksander Alekseev <aleksander@timescale.com> writes:
> Personally I like the change. It makes the output more explicit. In my
> use cases not running TAP tests typically is not something I want . So
> I would appreciate being warned with a long list of bright yellow
> "SKIP" messages. If I really want to skip TAP tests these messages are
> just informative and don't bother me.

+1 for counting such tests as "skipped" in the summary.  -1 for
emitting a message per skipped test.  If I'm intentionally not
running those tests, that would be very annoying noise, and
potentially would obscure messages I actually need to see.

(And about -100 for emitting such messages in yellow.  Doesn't
anybody who codes this stuff have a clue about vision problems?)

			regards, tom lane

  4. Re: Explicitly skip TAP tests under Meson if disabled

    Tristan Partin <tristan@neon.tech> — 2023-10-31T16:03:12Z

    Hi Peter,

You may find value in this Meson PR[0] adding a skip keyword argument to 
Meson's test() function. From what I understand of the PR and your 
issue, they seem related. If you could provide a comment describing why 
this is valuable to you, it would be good to help the Meson 
maintainers understand the use case better.

Thanks!

[0]: https://github.com/mesonbuild/meson/pull/12362

-- 
Tristan Partin
Neon (https://neon.tech)

  5. Re: Explicitly skip TAP tests under Meson if disabled

    Peter Eisentraut <peter@eisentraut.org> — 2023-11-02T13:52:14Z

    On 30.10.23 10:12, Tom Lane wrote:
> +1 for counting such tests as "skipped" in the summary.  -1 for
> emitting a message per skipped test.  If I'm intentionally not
> running those tests, that would be very annoying noise, and
> potentially would obscure messages I actually need to see.

In my usage, those messages only show up in the logs, not during a 
normal test run.  This is similar to other skip messages, like "skipped 
on Windows" or "skipped because LDAP not enabled" etc.

  6. Re: Explicitly skip TAP tests under Meson if disabled

    Andres Freund <andres@anarazel.de> — 2023-11-04T00:51:05Z

    Hi,

On 2023-10-30 05:45:52 -0400, Peter Eisentraut wrote:
> Under Meson, it is not very easy to see if TAP tests have been enabled or
> disabled, if you rely on the default auto setting.  You either need to
> carefully study the meson setup output, or you notice, what a minute, didn't
> there use to be like 250 tests, not only 80?
> 
> I think it would be better if we still registered the TAP tests in Meson
> even if the tap_tests option is disabled, but with a dummy command that
> registers them as skipped.  That way you get a more informative output like

Hm, ok. I've never felt I needed this, but I can see the point.


> See attached patch for a possible implementation.  (This uses perl as a hard
> build requirement.  We are planning to do that anyway, but obviously other
> implementations, such as using python, would also be possible.)

There's already other hard dependencies on perl in the meson build (generating
kwlist etc). We certainly error out if it's not available.


>  
> -        test(test_dir['name'] / onetap_p,
> -          python,
> -          kwargs: test_kwargs,
> -          args: testwrap_base + [
> -            '--testgroup', test_dir['name'],
> -            '--testname', onetap_p,
> -            '--', test_command,
> -            test_dir['sd'] / onetap,
> -          ],
> -        )
> +        if tap_tests_enabled
> +          test(test_dir['name'] / onetap_p,
> +            python,
> +            kwargs: test_kwargs,
> +            args: testwrap_base + [
> +              '--testgroup', test_dir['name'],
> +              '--testname', onetap_p,
> +              '--', test_command,
> +              test_dir['sd'] / onetap,
> +            ],
> +          )
> +        else
> +          test(test_dir['name'] / onetap_p,
> +               perl,
> +               args: ['-e', 'print "1..0 # Skipped: TAP tests not enabled"'],
> +               kwargs: test_kwargs)
> +        endif

I'd just use a single test() invocation here, and add an argument to testwrap
indicating that it should print out the skipped message. That way we a) don't
need two test() invocations, b) could still see the test name etc in the test
invocation.

Greetings,

Andres Freund

  7. Re: Explicitly skip TAP tests under Meson if disabled

    Peter Eisentraut <peter@eisentraut.org> — 2023-11-06T16:46:23Z

    On 04.11.23 01:51, Andres Freund wrote:
> I'd just use a single test() invocation here, and add an argument to testwrap
> indicating that it should print out the skipped message. That way we a) don't
> need two test() invocations, b) could still see the test name etc in the test
> invocation.

Is testwrap only meant to be used with the tap protocol mode of meson's 
test()?  Otherwise, this skip option would have produce different output 
for different protocols.

  8. Re: Explicitly skip TAP tests under Meson if disabled

    Andres Freund <andres@anarazel.de> — 2023-11-06T22:03:08Z

    Hi,

On 2023-11-06 17:46:23 +0100, Peter Eisentraut wrote:
> On 04.11.23 01:51, Andres Freund wrote:
> > I'd just use a single test() invocation here, and add an argument to testwrap
> > indicating that it should print out the skipped message. That way we a) don't
> > need two test() invocations, b) could still see the test name etc in the test
> > invocation.
> 
> Is testwrap only meant to be used with the tap protocol mode of meson's
> test()?  Otherwise, this skip option would have produce different output for
> different protocols.

Since Daniel added tap support to pg_regress it's only used with tap. If we
add something else, we can add a format parameter?

Greetings,

Andres Freund

  9. Re: Explicitly skip TAP tests under Meson if disabled

    Peter Eisentraut <peter@eisentraut.org> — 2023-11-15T10:02:19Z

    On 04.11.23 01:51, Andres Freund wrote:
> I'd just use a single test() invocation here, and add an argument to testwrap
> indicating that it should print out the skipped message. That way we a) don't
> need two test() invocations, b) could still see the test name etc in the test
> invocation.

Here is a patch that does it that way.

  10. Re: Explicitly skip TAP tests under Meson if disabled

    Andres Freund <andres@anarazel.de> — 2023-11-15T16:50:08Z

    On 2023-11-15 11:02:19 +0100, Peter Eisentraut wrote:
> On 04.11.23 01:51, Andres Freund wrote:
> > I'd just use a single test() invocation here, and add an argument to testwrap
> > indicating that it should print out the skipped message. That way we a) don't
> > need two test() invocations, b) could still see the test name etc in the test
> > invocation.
> 
> Here is a patch that does it that way.

WFM!