Stefan, Stefan Seefeld wrote:
Douglas Gregor wrote:
I didn't find any answer when I asked the last time, so I'm asking again:
FWIW I tried to answer the question before.
What is the meaning of the absolute number of 'regressions' ? Did this number really go up from the last report to the current one ? At least some 'new' ones stem from the inclusion of the 'gcc-4.1.1_sunos_i86pc' test run in this report, which wasn't present in the last.
In isolation that number isn't worth much. That being said I am a little puzzled by your fixation on this number. I think that the number together with the regression tables provides valuable information. It just requires some interpretation.
What determines the test runs that make it into a report ?
Availability. We rely on volunteers to provide test results.
Is this sunos platform really a primary platform for this release ? Why wasn't it tested before ?
Yes. At the point the platform was chosen we had frequent results. Testers drop in/out for all kinds of reasons. Hardware/vacation/job/configuration it's hard to blame a volunteer for any of this, but in the end we have a major reliability and turnaround issue. I would drop the platform right away but in this case we are dealing with a general python issue. Dropping the platform won't make it go away.
How are we ever going to get the number of unexpected failures down to zero ?
Fixing bugs?
I honestly don't believe it will ever happen, if we continue like that. :-(
Well in some way we already gave up. The only thing we are waiting on is the python stuff to get into shape.
May I suggest to fix a number of 'primary platforms' (and that may well translate to specific testers, at this point in the release process), and just disregard anything else ?
Personally I think we'll have to do something like this in the future. Right now we don't have the infrastructure in place to do this and the last thing I want to do at this point is fiddle with regression test infrastructure. Thomas -- Thomas Witt witt@acm.org