Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

Hello Heikki,

> I stared at the new test case for a while, and I must say it looks very 
> cryptic. It's not exactly this patch's fault - all the pgbench tests are 
> cryptic -

Perl is cryptic. Regexprs are cryptic.

> but I think we need to do something about that before adding any more 
> tests. I'm not sure what exactly, but I'd like them to be more like 
> pg_regress tests, where you have an expected output and you compare it 
> with the actual output. I realize that's not easy, because there are a 
> lot of varying numbers in the output, but we've got to do something.
>
> As a good first step, I wish the pgbench() function used named 
> arguments. [...]
>
> You would have something like this:
>
> my $elapsed = pgbench(
>  test_name => 'pgbench progress',
>  opts => '-T 2 -P 1 -l --aggregate-interval=1'

I do not like them much in perl because it changes the code significantly, 
but why not. That would be another patch anyway.

A lighter but efficient option would be to add a few comments on the 
larger calls, see attached.

> My other complaint about the new test is that it does nothing to check 
> if the output looks sensible. That's even harder to test, so it's 
> probably not worth the trouble to try. But as it is, how much value does 
> the test really have?

I do not understand. The list of expected regexpr on stdout and stderr 
*are* the checks out the outputs, and there quite a few plenty of them?

> It would fail, if --progress caused pgbench to crash, or if no progress 
> reports were printed at all, but I can't get very excited about that.

I cannot help you on this one: tests are never exciting:-)

The point is to improve code coverage, inducing that if a changes breaks 
something the test should help notice before field reports.

There is nothing "exciting" about that, indeed.

The current test status is that one could write a dozen abort() in the 
code and it would pass the tests.

Current tests coverage is much too low all over the project, see 
https://coverage.postgresql.org/ which looks abysmal to me. I would to put 
pgbench in the green. I do think that the whole project should be in the 
green when all tests are run. Although coverage is not everything there is 
about testing, it is a good start.

-- 
Fabien.