State of regression tests
As everybody noticed, the state of Boost regression tests is pretty chaotic at the moment. We have thousands of error reports, some of them can be tracked to issues in regression testing tools, and some of them can be tracked to issues in tester's configurations. It's completely impossible to do anything with 2000 error reports without some organized effort. Can we all agree on the following: 1. No commits to libraries code on RC branch whatsoever. 2. All patches to regression testing tools and Boost.Build code on RC should be posted to main boost mailing list, so that everybody knows what's being changed. 3. No changes in tester's configurations whatsoever, unless explicitly asked for. 4. No new testers whatsoever. I think that with those agreements, it would take me less than a week to stabilize regression results from Martin Wille (who is historically known to have no local misconfigurations), and knowing that regression system itself works, we can effectively decide what to do with other issues -- including those showing up in configurations that were not previously tested. Opinions? - Volodya
Vladimir Prus wrote:
3. No changes in tester's configurations whatsoever, unless explicitly asked for.
Opinions?
I think there may be a number of things that could be enhanced, in the long run (i.e. *after* the 'upcoming' release): * It would be good if the configurations were provided from a central place, to cut down on the parameters that impact the overall outcome. The buildbot project may help to organize this a little better. * More test-run metadata need to be stored and reported back, to allow the report generator to eliminate duplicate test run results, and to be able to make failures reproducible. Just some food for thought. Stefan -- ...ich hab' noch einen Koffer in Berlin...
Vladimir Prus wrote:
3. No changes in tester's configurations whatsoever, unless explicitly asked for.
Ok. As I heave just learned, running release tests was not so a good idea. Should i drop them, i.e. stop running and remove the results from the server?
4. No new testers whatsoever.
As I am running only since a week. Should I stay or should I go?
I think that with those agreements, it would take me less than a week to stabilize regression results from Martin Wille (who is historically known to have no local misconfigurations), and knowing that regression system itself works, we can effectively decide what to do with other issues -- including those showing up in configurations that were not previously tested.
Opinions?
Seems reasonable. Roland
Roland Schwarz wrote:
Vladimir Prus wrote:
3. No changes in tester's configurations whatsoever, unless explicitly asked for.
Ok. As I heave just learned, running release tests was not so a good idea. Should i drop them, i.e. stop running and remove the results from the server?
I believe so.
4. No new testers whatsoever.
As I am running only since a week. Should I stay or should I go?
Thanks for asking! My primary point was that we should avoid anything that uncontrollably change current picture, to avoid overlapping several factors. You're already in, so as soon as you don't change anything at all, you're neither increasing nor decreasing the number of reported errors, and so don't make it harder to track progress, so there's no need to stop running at this point. Thanks, Volodya
Hi, Vladimir Prus wrote:
As everybody noticed, the state of Boost regression tests is pretty chaotic at the moment. We have thousands of error reports, some of them can be tracked to issues in regression testing tools, and some of them can be tracked to issues in tester's configurations. It's completely impossible to do anything with 2000 error reports without some organized effort.
Agreed.
Can we all agree on the following:
1. No commits to libraries code on RC branch whatsoever.
2. All patches to regression testing tools and Boost.Build code on RC should be posted to main boost mailing list, so that everybody knows what's being changed.
Please add a cc to the testing list.
3. No changes in tester's configurations whatsoever, unless explicitly asked for.
Vladimir are you going to be the person in charge of this? I.e. *no* changes unless you ask for them? I think it's important that there is one person.
4. No new testers whatsoever.
I think that with those agreements, it would take me less than a week to stabilize regression results from Martin Wille (who is historically known to have no local misconfigurations), and knowing that regression system itself works, we can effectively decide what to do with other issues -- including those showing up in configurations that were not previously tested.
If it's not clear by now I completely agree with this strategy. Thanks Vladimir for taking care of this. Thomas Release Manager -- Thomas Witt witt@acm.org
participants (4)
-
Roland Schwarz -
Stefan Seefeld -
Thomas Witt -
Vladimir Prus