Horton

Thread

  1. Re: [HACKERS] Current sources?

    Tom Lane <tgl@sss.pgh.pa.us> — 1998-05-26T16:25:48Z

    Bruce Momjian <maillist@candle.pha.pa.us> writes:
> The author did test with the regression tests.  In fact, the
> regression tests are not up-to-date, so there are meny diffs even when
> the code works, and we can't expect someone to keep the regression
> tests spotless at all times.

Actually, I sympathize with David on this: I got burnt the same way
just a couple weeks ago.  (I blithely assumed that the regression tests
would test copy in/out ... they don't ...)

Perhaps the real lesson to be learned is that a little more effort
should be expended on the regression tests.  I have a couple of
suggestions:

1. As far as I've seen there is no documentation on how to create
   regression tests.  This should be documented and made as easy as
   possible, to encourage people to create tests for missing cases.

2. System variations (roundoff error differences, etc) create spurious
   test complaints that make it hard to interpret the results properly.
   Can anything be done to clean this up?

3. The TODO list should maintain a section on missing regression tests;
   any failure that gets by the regression tests should cause an entry
   to get made here.  This list would have a side benefit of warning
   developers about areas that are not getting tested, so that they know
   they have to do some hand testing if they change relevant code.

We can start the new TODO section with:

* Check destroydb.  (Currently, running regression a second time checks
  this, but a single run in a clean tree won't.)
* Check copy from stdin/to stdout.
* Check large-object interface.

What else have people been burnt by lately?

			regards, tom lane

  2. Re: [HACKERS] Current sources?

    Marc G. Fournier <scrappy@hub.org> — 1998-05-26T17:19:27Z

    On Tue, 26 May 1998, Tom Lane wrote:

> Bruce Momjian <maillist@candle.pha.pa.us> writes:
> > The author did test with the regression tests.  In fact, the
> > regression tests are not up-to-date, so there are meny diffs even when
> > the code works, and we can't expect someone to keep the regression
> > tests spotless at all times.
> 
> Actually, I sympathize with David on this: I got burnt the same way
> just a couple weeks ago.  (I blithely assumed that the regression tests
> would test copy in/out ... they don't ...)
> 
> Perhaps the real lesson to be learned is that a little more effort
> should be expended on the regression tests.  I have a couple of
> suggestions:
> 
> 1. As far as I've seen there is no documentation on how to create
>    regression tests.  This should be documented and made as easy as
>    possible, to encourage people to create tests for missing cases.
> 
> 2. System variations (roundoff error differences, etc) create spurious
>    test complaints that make it hard to interpret the results properly.
>    Can anything be done to clean this up?

	See the expected/int2-FreeBSD.out and similar files...I've done
what I can with the 'spurious test complaints...

  3. Re: [HACKERS] Current sources?

    David Gould <dg@illustra.com> — 1998-05-26T17:52:40Z

    Tom Lane:
> Perhaps the real lesson to be learned is that a little more effort
> should be expended on the regression tests.  I have a couple of
> suggestions:
> 
> 1. As far as I've seen there is no documentation on how to create
>    regression tests.  This should be documented and made as easy as
>    possible, to encourage people to create tests for missing cases.

Excellent idea. If everyone making a new feature could also make a test case
for it this would help us keep the system stable.

> 2. System variations (roundoff error differences, etc) create spurious
>    test complaints that make it hard to interpret the results properly.
>    Can anything be done to clean this up?

Hmmm, perhaps we could modify the tests to display results through a function
that rounded to the expected precision eg:

instead of

   select floatcol, doublecol from testtab;

use
   select display(floatcol, 8), display(doublecol, 16) from testtab;

 
> 3. The TODO list should maintain a section on missing regression tests;
>    any failure that gets by the regression tests should cause an entry
>    to get made here.  This list would have a side benefit of warning
>    developers about areas that are not getting tested, so that they know
>    they have to do some hand testing if they change relevant code.
> 
> We can start the new TODO section with:
> 
> * Check destroydb.  (Currently, running regression a second time checks
>   this, but a single run in a clean tree won't.)
> * Check copy from stdin/to stdout.
> * Check large-object interface.
> 
> What else have people been burnt by lately?

The int2, oidint2, int4, and oidint4 tests (and some others I think) are
currently failing because the text of a couple error messages changed and
the "expected" output was not updated. This kind of thing is pretty annoying
as whoever changed the messages really should have updated the tests as well.

If the current messages are preferred to the old messages, I will fix the test
output to match, although personally, I like the old messages better.

I will argue once again for a clean snapshot that is known to pass regression.
This snapshot could be just a CVS tag, but it is important when starting work
on a complex change to be able to know that any problems you have when you
are done are due to your work, not some pre-existing condition.

-dg

David Gould            dg@illustra.com           510.628.3783 or 510.305.9468 
Informix Software  (No, really)         300 Lakeside Drive  Oakland, CA 94612
"I believe OS/2 is destined to be the most important operating
system, and possibly program, of all time" - Bill Gates, Nov, 1987.

  4. Re: [HACKERS] Current sources?

    Thomas Lockhart <lockhart@alumni.caltech.edu> — 1998-05-27T02:22:05Z

    > > Perhaps the real lesson to be learned is that a little more effort
> > should be expended on the regression tests.  I have a couple of
> > suggestions:
> > 1. As far as I've seen there is no documentation on how to create
> >    regression tests.  This should be documented and made as easy as
> >    possible, to encourage people to create tests for missing cases.

Hmm. It ain't hard, but afaik the only people who have pushed on the
regression tests are scrappy and myself. We went for years with no
updates to the regression tests at all, and now have a somewhat stable
set of tests which actually measure many features of a s/w build.

> Excellent idea. If everyone making a new feature could also make a
> test case for it this would help us keep the system stable.

This would seem to be a truism. Any takers??

> > 2. System variations (roundoff error differences, etc) create 
> > spurious test complaints that make it hard to interpret the results 
> > properly. Can anything be done to clean this up?
> Hmmm, perhaps we could modify the tests to display results through a 
> function that rounded to the expected precision eg:
>    select display(floatcol, 8), display(doublecol, 16) from testtab;

Gee, maybe we should go back to the original v4.2/v1.0.x behavior of
rounding _all_ double precision floating point results to 6 digits :(

We've worked hard to get all of the regression tests to match at least
one platform (at the moment, Linux/i686) and scrappy has extended the
test mechanism to allow for platform-specific differences. But we don't
have access to all of the supported platforms, so others will need to
help (and they have been, at least some).

> > 3. The TODO list should maintain a section on missing regression 
> > tests; any failure that gets by the regression tests should cause an 
> > entry to get made here. This list would have a side benefit of 
> > warning developers about areas that are not getting tested, so that 
> > they know they have to do some hand testing if they change relevant 
> > code.

imho it will take more effort to maintain a todo list than to just
submit a patch for testing. I would be happy to maintain the "expected"
output if people would suggest new tests (and better, submit patches for
the test).

> I will argue once again for a clean snapshot that is known to pass 
> regression.

I snapshot the system all the time, and then do development for a week
or two or more on that revlocked snapshot. afaik the failures in
regression testing at the time I submitted my last "typing" patches were
due to differences in type conversion behavior; I didn't want the
changed behavior to become formalized until others had had a chance to
test and comment. (btw, no one has, and anyway I'm changing the results
for most of those back to what it was before).

It's pretty clear that many patches are submitted without full
regression testing; either way it would be helpful to post a comment
with patches saying how the patches affect the regression tests, or that
no testing was done. I'd like to see another person test patches before
committing to the source tree, but others might like to see where the
patches/changes are heading even before that so I can see arguments both
ways.

As has been suggested by yourself and others, regression test
contributions would be very helpful; so far the discussion amounts to
asking scrappy and myself to do _more_ work on the regression tests. I'd
like to see someone offering specific help at some point :)

Anyway, if Marc or I led this discussion you will probably just get more
ideas similar to what is already there; more brainstorming on the
hackers list from y'all will lead to some good new ideas so I'll shut up
now...

                     - Tom

  5. Re: [HACKERS] Current sources?

    Mattias Kregert <matti@algonet.se> — 1998-05-27T11:34:01Z

    Tom Lane wrote:
> 
> 2. System variations (roundoff error differences, etc) create spurious
>    test complaints that make it hard to interpret the results properly.
>    Can anything be done to clean this up?
> 

It would be good if the backend looked at errno and put out an
appropriate sqlcode number and a human readable message, instead
of using the system error messages.
That would eliminate some of the regression test diff output.

/* m */

  6. Re: [HACKERS] Current sources?

    David Gould <dg@illustra.com> — 1998-05-31T08:00:00Z

    The Hermit Hacker wrote: 
> On Tue, 26 May 1998, Tom Lane wrote:
> > Perhaps the real lesson to be learned is that a little more effort
> > should be expended on the regression tests.  I have a couple of
> > suggestions:
> > 
> > 1. As far as I've seen there is no documentation on how to create
> >    regression tests.  This should be documented and made as easy as
> >    possible, to encourage people to create tests for missing cases.
> > 
> > 2. System variations (roundoff error differences, etc) create spurious
> >    test complaints that make it hard to interpret the results properly.
> >    Can anything be done to clean this up?
> 
> 	See the expected/int2-FreeBSD.out and similar files...I've done
> what I can with the 'spurious test complaints...
> 

Thanks. One question, is there any reason we can't use the intx tests on
all the platforms? I realize that float it another set of problems, but it
seems that int should be be the same?

-dg

David Gould           dg@illustra.com            510.628.3783 or 510.305.9468
Informix Software                      300 Lakeside Drive   Oakland, CA 94612
 - A child of five could understand this!  Fetch me a child of five.

  7. Re: [HACKERS] Current sources?

    Marc G. Fournier <scrappy@hub.org> — 1998-05-31T17:40:54Z

    On Sun, 31 May 1998, David Gould wrote:

> Thanks. One question, is there any reason we can't use the intx tests on
> all the platforms? I realize that float it another set of problems, but it
> seems that int should be be the same?

10c10
< ERROR:  pg_atoi: error reading "100000": Result too large
---
> ERROR:  pg_atoi: error reading "100000": Math result not representable

The changes are more error message relatd then anything...

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org

  8. Re: [HACKERS] Current sources?

    David Gould <dg@illustra.com> — 1998-05-31T18:36:27Z

    > On Sun, 31 May 1998, David Gould wrote:
> 
> > Thanks. One question, is there any reason we can't use the intx tests on
> > all the platforms? I realize that float it another set of problems, but it
> > seems that int should be be the same?
> 
> 10c10
> < ERROR:  pg_atoi: error reading "100000": Result too large
> ---
> > ERROR:  pg_atoi: error reading "100000": Math result not representable
> 
> The changes are more error message relatd then anything...
> 
> Marc G. Fournier                                
> Systems Administrator @ hub.org 
> primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 

Thats what I thought. So can we just rename the int*-*BSD.out to int*.out?
-dg

  9. Re: [HACKERS] Current sources?

    Marc G. Fournier <scrappy@hub.org> — 1998-05-31T18:42:46Z

    On Sun, 31 May 1998, David Gould wrote:

> > On Sun, 31 May 1998, David Gould wrote:
> > 
> > > Thanks. One question, is there any reason we can't use the intx tests on
> > > all the platforms? I realize that float it another set of problems, but it
> > > seems that int should be be the same?
> > 
> > 10c10
> > < ERROR:  pg_atoi: error reading "100000": Result too large
> > ---
> > > ERROR:  pg_atoi: error reading "100000": Math result not representable
> > 
> > The changes are more error message relatd then anything...
> > 
> > Marc G. Fournier                                
> > Systems Administrator @ hub.org 
> > primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 
> 
> Thats what I thought. So can we just rename the int*-*BSD.out to int*.out?

	No cause then that will break the Linux regression tests :)

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org

  10. Re: [HACKERS] Current sources?

    Thomas Lockhart <lockhart@alumni.caltech.edu> — 1998-05-31T19:01:07Z

    > > > > Is there any reason we can't use the intx tests on all the 
> > > > platforms?
> > > The changes are more error message relatd then anything...
> > So can we just rename the int*-*BSD.out to int*.out?
> No cause then that will break the Linux regression tests :)

... which has been the regression reference platform since scrappy and I
resurrected the regression test suite 'bout a year ago for v6.1...

I assume that most platforms have some differences. Or would we find
lots more matching each other if we chose something other than Linux for
the reference output?

                     - Tom

  11. Re: [HACKERS] Current sources?

    David Gould <dg@illustra.com> — 1998-05-31T22:07:45Z

     Marc G. Fournier wrote:
> > > On Sun, 31 May 1998, David Gould wrote:
> > > 
> > > > Thanks. One question, is there any reason we can't use the intx tests on
> > > > all the platforms? I realize that float it another set of problems, but it
> > > > seems that int should be be the same?
> > > 
> > > 10c10
> > > < ERROR:  pg_atoi: error reading "100000": Result too large
> > > ---
> > > > ERROR:  pg_atoi: error reading "100000": Math result not representable
> > > 
> > > The changes are more error message relatd then anything...
> > > 
> > Thats what I thought. So can we just rename the int*-*BSD.out to int*.out?
> 
> 	No cause then that will break the Linux regression tests :)

For the "int" tests?  I hope not. Anyhow, I will test this and see if I can
clean up some regression issues. I plan to break a lot of stuff in the next
few weeks (ok, months) and sure want to be able to count on the regression
suite to help me find my way.

-dg

  12. Re: [HACKERS] Current sources?

    David Gould <dg@illustra.com> — 1998-05-31T22:13:15Z

    > > > > > Is there any reason we can't use the intx tests on all the 
> > > > > platforms?
> > > > The changes are more error message relatd then anything...
> > > So can we just rename the int*-*BSD.out to int*.out?
> > No cause then that will break the Linux regression tests :)
> 
> ... which has been the regression reference platform since scrappy and I
> resurrected the regression test suite 'bout a year ago for v6.1...
> 
> I assume that most platforms have some differences. Or would we find
> lots more matching each other if we chose something other than Linux for
> the reference output?
> 
>                      - Tom

Hmmm, I find that I get lots of diffs on the floating point tests as
I am running Linux with the new "glibc". I suspect the reference platform
is the old "libc5" Linux. We might want to move the reference to "glibc" 
Linux as this will be the majority plaform very soon. And since glibc is
the not just for Linux it will even help with other platforms in the comming
few months.

Anyway, I am willing to work on the tests a little but do not want to "take
them over". Who "owns" them now, perhaps I could co-ordinate and ask for
advice from that person?

-dg

David Gould            dg@illustra.com           510.628.3783 or 510.305.9468 
Informix Software  (No, really)         300 Lakeside Drive  Oakland, CA 94612
"Of course, someone who knows more about this will correct me if I'm wrong,
 and someone who knows less will correct me if I'm right."
               --David Palmer (palmer@tybalt.caltech.edu)

  13. Re: [HACKERS] Current sources?

    Marc G. Fournier <scrappy@hub.org> — 1998-05-31T22:25:02Z

    On Sun, 31 May 1998, David Gould wrote:

> Anyway, I am willing to work on the tests a little but do not want to "take
> them over". Who "owns" them now, perhaps I could co-ordinate and ask for
> advice from that person?

	Nobody "owns" them...Thomas and I have tried to keep them
relatively up to date, with Thomas doing the most part of the work on a
Linux platform...

	Stuff like the int* test 'expected' output files are generated
under Linux, which generates a different error message then the same
test(s) under FreeBSD/NetBSD :(

  14. Re: [HACKERS] Current sources?

    David Gould <dg@illustra.com> — 1998-05-31T23:17:33Z

    > On Sun, 31 May 1998, David Gould wrote:
> 
> > Anyway, I am willing to work on the tests a little but do not want to "take
> > them over". Who "owns" them now, perhaps I could co-ordinate and ask for
> > advice from that person?
> 
> 	Nobody "owns" them...Thomas and I have tried to keep them
> relatively up to date, with Thomas doing the most part of the work on a
> Linux platform...
> 
> 	Stuff like the int* test 'expected' output files are generated
> under Linux, which generates a different error message then the same
> test(s) under FreeBSD/NetBSD :(

Ok, now I am confused. Isn't the error message "our" error message? If so,
can't we make it the same?

-dg

  15. Re: [HACKERS] Current sources?

    Thomas Lockhart <lockhart@alumni.caltech.edu> — 1998-06-01T06:29:55Z

    > > Nobody "owns" them...Thomas and I have tried to keep them
> > relatively up to date, with Thomas doing the most part of the work 
> > on a Linux platform...
> > Stuff like the int* test 'expected' output files are generated
> > under Linux, which generates a different error message then the same
> > test(s) under FreeBSD/NetBSD :(
> Ok, now I am confused. Isn't the error message "our" error message? If 
> so, can't we make it the same?

Nope. Some messages come from the system apparently. I can't remember
how they come about, but the differences are not due to #ifdef FreeBSD
blocks in the code :)

The only differences I know of in the regression tests are due to
numeric rounding, math libraries and system error messages.

I will point out that although no one really "owns" the regression tests
(in the spirit that everyone can and should contribute) I (and others)
have run them extensively in support of releases. It is important that
whoever is running the "reference platform" be willing to run regression
tests ad nauseum, and to track down any problems. I've done so the last
few releases.

When/if this doesn't happen, we get a flakey release.

                   - Tom

  16. Re: [HACKERS] Current sources?

    David Gould <dg@illustra.com> — 1998-06-02T06:01:14Z

    > > > Nobody "owns" them...Thomas and I have tried to keep them
> > > relatively up to date, with Thomas doing the most part of the work 
> > > on a Linux platform...
> > > Stuff like the int* test 'expected' output files are generated
> > > under Linux, which generates a different error message then the same
> > > test(s) under FreeBSD/NetBSD :(
> > Ok, now I am confused. Isn't the error message "our" error message? If 
> > so, can't we make it the same?
> 
> Nope. Some messages come from the system apparently. I can't remember
> how they come about, but the differences are not due to #ifdef FreeBSD
> blocks in the code :)

Thank goodness! I always worry about that when dealing with *BSD people ;-)
 
> The only differences I know of in the regression tests are due to
> numeric rounding, math libraries and system error messages.

That is about what I see. 

> I will point out that although no one really "owns" the regression tests
> (in the spirit that everyone can and should contribute) I (and others)
> have run them extensively in support of releases. It is important that
> whoever is running the "reference platform" be willing to run regression
> tests ad nauseum, and to track down any problems. I've done so the last
> few releases.

Ok, I will make a set of Linux glibc expected files for 6.3.2 and if that
works send them in. Not sure how to handle the reference Linux vs glibc
Linux issue in terms of the way the tests are structured and platforms named,
but they do have different rounding behavior and messages.

Of course, someone is welcome to beat me to this, no really, go ahead...

-dg

David Gould            dg@illustra.com           510.628.3783 or 510.305.9468 
Informix Software  (No, really)         300 Lakeside Drive  Oakland, CA 94612
"Don't worry about people stealing your ideas.  If your ideas are any
 good, you'll have to ram them down people's throats." -- Howard Aiken

  17. Re: [HACKERS] Current sources?

    Thomas Lockhart <lockhart@alumni.caltech.edu> — 1998-06-02T13:56:17Z

    > > It is important that
> > whoever is running the "reference platform" be willing to run 
> > regression tests ad nauseum, and to track down any problems.
> Ok, I will make a set of Linux glibc expected files for 6.3.2 and if 
> that works send them in. Not sure how to handle the reference Linux vs 
> glibc Linux issue in terms of the way the tests are structured and 
> platforms named, but they do have different rounding behavior and 
> messages.

I'm running RH5.0 at work, but have RH4.2 at home. I'm reluctant to
upgrade at home because I have _all_ Postgres releases from v1.0.9 to
current installed and I can fire them up for testing in less than a
minute. If I upgrade to the new glibc2, I might have trouble rebuilding
the old source trees. Anyway, will probably upgrade sometime in the next
few months, and then the reference platform will be glibc2-based.

If you are generating new "expected" files for glibc2 shouldn't they be
based on the current development tree? Or are you providing them as a
patch for v6.3.2 to be installed in /pub/patches??

                   - Tom

  18. Re: [HACKERS] Current sources?

    Marc G. Fournier <scrappy@hub.org> — 1998-06-03T21:26:46Z

    On Mon, 1 Jun 1998, David Gould wrote:

> > Nope. Some messages come from the system apparently. I can't remember
> > how they come about, but the differences are not due to #ifdef FreeBSD
> > blocks in the code :)
> 
> Thank goodness! I always worry about that when dealing with *BSD people ;-)

	*BSD people??  At least us *BSD people work at making sure that
software developed works on everyone's elses choice of OS too...not like
some Linux developers out there (ie. Wine) :)

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org

  19. Re: [HACKERS] Current sources?

    David Gould <dg@illustra.com> — 1998-06-04T03:18:11Z

    > On Mon, 1 Jun 1998, David Gould wrote:
> 
> > > Nope. Some messages come from the system apparently. I can't remember
> > > how they come about, but the differences are not due to #ifdef FreeBSD
> > > blocks in the code :)
> > 
> > Thank goodness! I always worry about that when dealing with *BSD people ;-)
> 
> 	*BSD people??  At least us *BSD people work at making sure that
> software developed works on everyone's elses choice of OS too...not like
> some Linux developers out there (ie. Wine) :)

I hear that they didn't do an AS/400 port either ;-).
-dg