At 08:30 AM 2/12/2004, David Abrahams wrote:
Beman Dawes <bdawes@acm.org> writes:
Anyhow, I think your point about multiple reporting is a good one. The volume of tests is just too high. Fewer, more comprehensive, tests would be easier to monitor. Also fewer compilers. Do we really need to test every version of GCC and VC++ for the last four years?
Yes, IMO, if people want to support those compilers, we do need to test them.
It comes down to resources. I suppose if people really care about older compilers, they will be willing to contribute testing.
If our testing was more focused, we could cycle the tests more often too.
I'm really unconvinced that we need smaller tests on fewer compilers.
I don't think smaller tests is a good idea. I'm asking for "more comprehensive" tests. Say a library now covers 100 test cases spread out among five test programs. I'd like to see those five programs refactored into one program, still covering the 100 test cases. Or even adding more test cases. A single program would cut the overhead, including the human overhead. It would be easier to know if any particular tests was a resource problem if there was a bjam option to time the build and run aspects of the tests. The change Jeremy made recently to one or two of the graph tests noticeably sped testing, but it was only happenstance that I realized they were problems.
Human monitoring is just too error-prone. Why risk it? Why not have comprehensive tests with automated notifications when something breaks? It seems to me that less testing can only result in fewer things working, and coupled with human monitoring it's just going to make things worse, not better.
Great! Let's give it a try. Who has to do what to start? --Beman