Thread

1. Re: Two weeks to feature freeze

   Dann Corbit <dcorbit@connx.com> — 2003-06-23T19:42:44Z

   > -----Original Message-----
   > From: scott.marlowe [mailto:scott.marlowe@ihs.com] 
   > Sent: Monday, June 23, 2003 12:25 PM
   > To: Dann Corbit
   > Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development
   > Subject: Re: [HACKERS] Two weeks to feature freeze
   > 
   > 
   > On Mon, 23 Jun 2003, Dann Corbit wrote:
   > 
   > > Vendor A: "We think our tool is pretty solid and our end 
   > users hardly 
   > > ever turn up any bugs."
   > > 
   > > Vendor B:" We think our tool is pretty solid and our 8500 tests 
   > > currently show only 3 defects with the released version, 
   > and these are 
   > > low impact issues.  To view our current database of issues, 
   > log onto 
   > > web form <page>."
   > > 
   > > Which tool would you prefer to install?
   > 
   > The one I've tested and found to meet my needs, both now and 
   > by providing 
   > fixes when I needed it.
   > 
   > Real world example:  We run Crystal Reports Enterprise 
   > edition where I 
   > work.  It's tested thouroughly (supposedly) and has all kinds of QA.  
   > However, getting it to work right and stay up is a nightmare. 
   >  It's taken 
   > them almost a year to get around to testing against the OpenLDAP LDAP 
   > server we use.  The box said "LDAP V3 compliant" and they 
   > assured us that 
   > it was.  Well, it doesn't work with our LDAP V3 compliant 
   > LDAP server at 
   > all, and the problem is something they can't fix for months 
   > because it 
   > doesn't fit into their test cycle.
   > 
   > 
   > Real world example: Postgresql aggregates in subselects. 
   > Someone found a bug in subselects in Postgresql with inner 
   > references to 
   > outter aggregates.  The postgresql team delivered a patch in 
   > less than a 
   > week.  User tested it and it works.
   > 
   > I'm not against testing and all, but as one of the many beta 
   > testers for 
   > Postgresql, I do feel a bit insulted by your attitude that only a 
   > cohesive, organized testing effort can result in a reliable product.
   
   Let me rephrase it:
   "Only a  cohesive, organized testing effort can result in a product that
   is proven reliable."
   
   Without such an effort, it is only an educated guess as to whether the
   product is reliable or not.  The data is the most valuable software
   component in an organization.  It is worth more than the hardware and it
   is worth more than the software.  If you are going to trust one billion
   dollars worth of corporate data on a software system, you ought to
   ensure that the system has been carefully tested.  I don't think that is
   just an opinion.  It's simply common sense.
    
   > I take my support of Postgresql seriously, and answer many 
   > questions every 
   > week in the general, sql, and performance mailing lists.  I'm 
   > not paid to 
   > do it, I stay at work an extra hour or so each day to "pay 
   > for it."  I 
   > test every beta and RC release on our dataset at work, and 
   > with a test 
   > load to make sure it works for us, and it does.
   > 
   > I offered to beta test for Crystal Reports and was told they weren't 
   > interested, they can do it in house.  Their support, like many big 
   > commercial houses, is designed more to make my boss's boss 
   > happy, not me, 
   > and it shows every day in how they fail to provide timely support for 
   > their product while playing CYA to the higherups.
   
   A long test cycle does result in a slower patch.  But when you get the
   patch, it is going to work and not introduce new problems.
   
   The resistance to testing is typical of programmers.  The PostgreSQL
   group is a group of programmers.  I don't think I can change anyone's
   mind, since the most significant people on the list don't think it is
   worth the bother.
   
   Therefore, I am going to stop harping on it.
   
   

2. Re: Two weeks to feature freeze

   Bruce Momjian <pgman@candle.pha.pa.us> — 2003-06-23T19:44:53Z

   Dann Corbit wrote:
   > Let me rephrase it:
   > "Only a  cohesive, organized testing effort can result in a product that
   > is proven reliable."
   > 
   > Without such an effort, it is only an educated guess as to whether the
   > product is reliable or not.  The data is the most valuable software
   > component in an organization.  It is worth more than the hardware and it
   > is worth more than the software.  If you are going to trust one billion
   > dollars worth of corporate data on a software system, you ought to
   > ensure that the system has been carefully tested.  I don't think that is
   > just an opinion.  It's simply common sense.
   
   True, it is an "educated guess", but it turns out our educated guess
   method produces more reliable code than the exhaustive testing method,
   so though there isn't as much of a _feeling_ of confidence, there is a
   _result_ of more reliability.
   
   -- 
     Bruce Momjian                        |  http://candle.pha.pa.us
     pgman@candle.pha.pa.us               |  (610) 359-1001
     +  If your life is a hard drive,     |  13 Roberts Road
     +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
   
   

3. Re: Two weeks to feature freeze

   scott.marlowe <scott.marlowe@ihs.com> — 2003-06-23T20:00:46Z

   On Mon, 23 Jun 2003, Dann Corbit wrote:
   
   > > -----Original Message-----
   > > From: scott.marlowe [mailto:scott.marlowe@ihs.com] 
   > > Sent: Monday, June 23, 2003 12:25 PM
   > > To: Dann Corbit
   > > Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development
   > > Subject: Re: [HACKERS] Two weeks to feature freeze
   > > 
   > > 
   > > On Mon, 23 Jun 2003, Dann Corbit wrote:
   > > 
   > > > Vendor A: "We think our tool is pretty solid and our end 
   > > users hardly 
   > > > ever turn up any bugs."
   > > > 
   > > > Vendor B:" We think our tool is pretty solid and our 8500 tests 
   > > > currently show only 3 defects with the released version, 
   > > and these are 
   > > > low impact issues.  To view our current database of issues, 
   > > log onto 
   > > > web form <page>."
   > > > 
   > > > Which tool would you prefer to install?
   > > 
   > > The one I've tested and found to meet my needs, both now and 
   > > by providing 
   > > fixes when I needed it.
   > > 
   > > Real world example:  We run Crystal Reports Enterprise 
   > > edition where I 
   > > work.  It's tested thouroughly (supposedly) and has all kinds of QA.  
   > > However, getting it to work right and stay up is a nightmare. 
   > >  It's taken 
   > > them almost a year to get around to testing against the OpenLDAP LDAP 
   > > server we use.  The box said "LDAP V3 compliant" and they 
   > > assured us that 
   > > it was.  Well, it doesn't work with our LDAP V3 compliant 
   > > LDAP server at 
   > > all, and the problem is something they can't fix for months 
   > > because it 
   > > doesn't fit into their test cycle.
   > > 
   > > 
   > > Real world example: Postgresql aggregates in subselects. 
   > > Someone found a bug in subselects in Postgresql with inner 
   > > references to 
   > > outter aggregates.  The postgresql team delivered a patch in 
   > > less than a 
   > > week.  User tested it and it works.
   > > 
   > > I'm not against testing and all, but as one of the many beta 
   > > testers for 
   > > Postgresql, I do feel a bit insulted by your attitude that only a 
   > > cohesive, organized testing effort can result in a reliable product.
   > 
   > Let me rephrase it:
   > "Only a  cohesive, organized testing effort can result in a product that
   > is proven reliable."
   
   No, it isn't proven reliable PERIOD, it's proven reliable under the exact 
   conditions of the testing procedure you've implemented.  And no matter how 
   idiot proof we try to make Postgresql and its testing, someone else will 
   always make a better idiot.  :-)  Actually, what I'm saying is that the 
   corner cases are the ones that are hard to predict, and no amount of 
   effort up front is going to find a corner case you haven't thought of yet.
   
   > Without such an effort, it is only an educated guess as to whether the
   > product is reliable or not.  The data is the most valuable software
   > component in an organization.  It is worth more than the hardware and it
   > is worth more than the software.  If you are going to trust one billion
   > dollars worth of corporate data on a software system, you ought to
   > ensure that the system has been carefully tested.  I don't think that is
   > just an opinion.  It's simply common sense.
   
   But if that is true, then Postgresql should cause me no end of problems as 
   it crashes down around my feet every other week.  Oddly, the dbas for the 
   other systems here at work (Oracle and MSSQL server) have a much higher 
   workload keeping their factory tested databases up and running.  In over 
   four years of use, we have had exactly ZERO downtime of postgresql.
   
   Carefully testing the system is what I, the DBA of our postgresql servers, 
   do.  Only I know how we use the system, so no matter how you or Bruce or 
   Tom might test it, I'll always be able to do something you wouldn't think 
   of, because you're simply not in my shoes.
   
   It's not an educated guess that postgresql works for us, it's proven over 
   and over again every single day by the throrough testing and use of every 
   Postgresql user.
   
   > > I take my support of Postgresql seriously, and answer many 
   > > questions every 
   > > week in the general, sql, and performance mailing lists.  I'm 
   > > not paid to 
   > > do it, I stay at work an extra hour or so each day to "pay 
   > > for it."  I 
   > > test every beta and RC release on our dataset at work, and 
   > > with a test 
   > > load to make sure it works for us, and it does.
   > > 
   > > I offered to beta test for Crystal Reports and was told they weren't 
   > > interested, they can do it in house.  Their support, like many big 
   > > commercial houses, is designed more to make my boss's boss 
   > > happy, not me, 
   > > and it shows every day in how they fail to provide timely support for 
   > > their product while playing CYA to the higherups.
   > 
   > A long test cycle does result in a slower patch.  But when you get the
   > patch, it is going to work and not introduce new problems.
   
   Nice theory, but it isn't provable by my experience.  While I've put .0 
   releases of postgresql into production many a times, usually with no 
   issues at all, I never have and never will put a .0 release of Crystal 
   Reports online.  They've taught me well not to trust their initial 
   release.
   
   > The resistance to testing is typical of programmers.  The PostgreSQL
   > group is a group of programmers.  I don't think I can change anyone's
   > mind, since the most significant people on the list don't think it is
   > worth the bother.
   
   No, I am NOT A postgresql programmer, I am a Postgresql user.  I do 
   program, but that's in support of my job as a PHP dev / database admin.  
   I've found myself looking through the postgresql code only an odd half 
   dozen times.  I've never NEEDED to hack it, as it seems to just work.
   
   > Therefore, I am going to stop harping on it.
   
   Good move.  You're busily shooting at ghosts right now.
   
   While there's always room for more testing, the best way to do it is to 
   roll up your sleaves and write your own tests or add to the regression 
   tests.  I've written quite a few load tests in Perl over the years to make 
   sure that postgresql can hold up to the kind of load we tend to throw at 
   it.  So have hundreds of other Postgresql users you've never met.  Just 
   because no one has a centralized plan and document for testing doesn't 
   mean it isn't getting done, it just means you can't see it all that well.
   
   I think of your testing methodology as the equivalent of the old Soviet 
   Union's Five year plans.  They were thorough and well documented, but 
   never worked as well as planned.  Postgresql is like the free market 
   system.  no one's completely in charge, and the more you try to control it 
   the more it tries to be free.  Ultimately, the chaos of the free market 
   won out.
   
   It may be reassuring to think your product is very well tested before it 
   goes out the door, but it's a false security, proven over and over by 
   commercial products that simply don't work in the field because of 
   problems that the original designers never envisioned, and now that they 
   have a thorough and long drawn out testing cycle, it simply takes longer 
   and longer to get fixes, while providing little, if any, improvement in 
   quality.
   
   
   

4. Re: Two weeks to feature freeze

   Nigel J. Andrews <nandrews@investsystems.co.uk> — 2003-06-23T20:29:47Z

   On Mon, 23 Jun 2003, Dann Corbit wrote:
   
   > > -----Original Message-----
   > > From: scott.marlowe [mailto:scott.marlowe@ihs.com] 
   > > Sent: Monday, June 23, 2003 12:25 PM
   > > To: Dann Corbit
   > > Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development
   > > Subject: Re: [HACKERS] Two weeks to feature freeze
   > > 
   > > 
   > > On Mon, 23 Jun 2003, Dann Corbit wrote:
   > > 
   > > > Vendor A: "We think our tool is pretty solid and our end 
   > > users hardly 
   > > > ever turn up any bugs."
   > > > 
   > > > Vendor B:" We think our tool is pretty solid and our 8500 tests 
   > > > currently show only 3 defects with the released version, 
   > > and these are 
   > > > low impact issues.  To view our current database of issues, 
   > > log onto 
   > > > web form <page>."
   > > > 
   > > > Which tool would you prefer to install?
   > > 
   > > The one I've tested and found to meet my needs, both now and 
   > > by providing 
   > > fixes when I needed it.
   
   How about the one that doesn't run tests in order to show how much better it is
   than the competition but to actually test operation? In other words Vendor B
   has an interest in having the tests pass, what gives you the confidence it just
   hasn't listed the ones that fail and that the tests that do pass are not just
   testing something vendor B wants to show it can do?
   
   
   > > Real world example:  We run Crystal Reports Enterprise 
   > > edition where I 
   > > work.  It's tested thouroughly (supposedly) and has all kinds of QA.  
   > > However, getting it to work right and stay up is a nightmare. 
   > >  It's taken 
   > > them almost a year to get around to testing against the OpenLDAP LDAP 
   > > server we use.  The box said "LDAP V3 compliant" and they 
   > > assured us that 
   > > it was.  Well, it doesn't work with our LDAP V3 compliant 
   > > LDAP server at 
   > > all, and the problem is something they can't fix for months 
   > > because it 
   > > doesn't fit into their test cycle.
   > > 
   > > 
   > > Real world example: Postgresql aggregates in subselects. 
   > > Someone found a bug in subselects in Postgresql with inner 
   > > references to 
   > > outter aggregates.  The postgresql team delivered a patch in 
   > > less than a 
   > > week.  User tested it and it works.
   > > 
   > > I'm not against testing and all, but as one of the many beta 
   > > testers for 
   > > Postgresql, I do feel a bit insulted by your attitude that only a 
   > > cohesive, organized testing effort can result in a reliable product.
   > 
   > Let me rephrase it:
   > "Only a  cohesive, organized testing effort can result in a product that
   > is proven reliable."
   > 
   > Without such an effort, it is only an educated guess as to whether the
   > product is reliable or not.  The data is the most valuable software
   > component in an organization.  It is worth more than the hardware and it
   > is worth more than the software.  If you are going to trust one billion
   > dollars worth of corporate data on a software system, you ought to
   > ensure that the system has been carefully tested.  I don't think that is
   > just an opinion.  It's simply common sense.
   
   So you've never worked on a project where the data is of high value, since in
   those circumstances the customer is always going to apply their own acceptance
   testing anyway. If you think that doesn't happen you try sitting through 2
   solid days of Y2k testing on _one_ system and tell me customers never do their
   own testing.
   
   
   > Therefore, I am going to stop harping on it.
   
   But there is no need to, as has been mentioned before, if the testing is not
   upto your level of testing submit something that makes it so. Having said that
   I do believe you mentioned that you didn't have the time to create something
   but you would be happy to test it, i.e. test the test.
   
   
   -- 
   Nigel J. Andrews
   
   
   

5. Re: Two weeks to feature freeze

   Lamar Owen <lamar.owen@wgcr.org> — 2003-06-23T21:59:48Z

   On Monday 23 June 2003 15:42, Dann Corbit wrote:
   > Let me rephrase it:
   > "Only a  cohesive, organized testing effort can result in a product that
   > is proven reliable."
   
   One can never 100% prove reliability without time in the field with real-world 
   data, testing or no testing.  I would dare say that there are numerous 
   reliable software packages out there in OSS-land that have never had that 
   sort of testing.  But it really hinges on ones definition of proof, no?
   
   Furthermore, the beta testers here in hackers are not 'end-users' per se.  The 
   people in hackers who test the betas are very valuable as our QA team.
   
   PostgreSQL is already proven reliable, to various degrees of reliability, 
   enough to where a PostgreSQL backend was approved as the new .ORG registry.  
   The track record continues, without mathematically rigorous and cohesive 
   testing.  Such testing would be useful, of course, by it is not required for 
   our purposes.  
   
   Those who want rigorous testing can do the rigorous testing.  You yourself 
   said that your company has a separate QA team from the development team; OK, 
   organize a rigorous QA team.  While you won't stop releases (unless you find 
   showstopper bugs, like those that have been found by our wonderful hackers 
   testers), your input into actually testing PostgreSQL (as opposed to trying 
   to convince someone else to test for you) would be valuable.  But you're not 
   going to get me to do it; I do enough testing of a varied nature already.  I 
   do this for fun.
   
   If you find testing fun, more power to you. :-)
   -- 
   Lamar Owen
   WGCR Internet Radio
   1 Peter 4:11
   
   
   

6. Re: Two weeks to feature freeze

   Marc G. Fournier <scrappy@postgresql.org> — 2003-06-24T01:43:57Z

   On Mon, 23 Jun 2003, Dann Corbit wrote:
   
   > The resistance to testing is typical of programmers.  The PostgreSQL
   > group is a group of programmers.  I don't think I can change anyone's
   > mind, since the most significant people on the list don't think it is
   > worth the bother.
   >
   > Therefore, I am going to stop harping on it.
   
   *rofl* I believe several of us have suggested that we would welcome
   submissions for improved testing ... so, what, we feel that the test that
   is done is sufficient, but are willing to accept submissions to improve
   it, but you aren't willing to spend the time/effort to do so?
   
   We might be a bunch of 'typical programmers', but you definitely sound
   like a typical "I want you to do something to make life easier for me, but
   am not willing to expend the time or effort to do anyting about it" ...
   
   
   

7. Re: Two weeks to feature freeze

   Jan Wieck <janwieck@yahoo.com> — 2003-06-24T05:09:56Z

   scott.marlowe wrote:
   > On Mon, 23 Jun 2003, Dann Corbit wrote:
    > [Dann Corbit wrote a lot]
    > [...]
   > It may be reassuring to think your product is very well tested before it 
   > goes out the door, but it's a false security, proven over and over by 
   > commercial products that simply don't work in the field because of 
   > problems that the original designers never envisioned, and now that they 
   > have a thorough and long drawn out testing cycle, it simply takes longer 
   > and longer to get fixes, while providing little, if any, improvement in 
   > quality.
   
   Scott, it's worse.
   
   It's been back in the early 90's, when we had WfW-3.11 systems with some 
   MS-Word dinosaur, and we just lost 14 days of work because it simply 
   crashed on loading the document. The Microsoft support solution was 
   something that lost all the formatting, indexing and cross references of 
   a structured 250 page concept. I don't remember the exact procedure as 
   my brain cells did overcharge, but the dummy on the hotline really 
   believed that their thoroughly tested software wasn't the problem and 
   that the error lies within our document. That that was a file, written 
   by their thoroughly tested software was a point she really didn't catch.
   
   This dumb hotline girl is the type of people, Dann Corbit's test 
   strategy will reassure. Plus maybe a few (unfortunately important but 
   otherwise useless) managers. Other than that, it'll not make the life of 
   the average DBA any better. Big amounts of useless tests just give 
   otherwise clueless people the false impression, the error must be 
   somewhere else. MySQL's crash-me is a perfect example for that.
   
   
   Jan
   
   -- 
   #======================================================================#
   # It's easier to get forgiveness for being wrong than for being right. #
   # Let's break this rule - forgive me.                                  #
   #================================================== JanWieck@Yahoo.com #