David Abrahams wrote:
"Robert Ramey" <ramey@rrsd.com> writes:
So if its not the same discussion but it is related. Of course we can test less. But the root of the problem is that probably only a small percentage of the effort invested in testing is actually testing anything. I know I've brought this up before but made no headway so I won't harp on it anymore.
It's hard to understand your objection to retesting library A in combination with a changed library B on which it depends, since that is essentially what you're doing with the various parts of the serialization with your N x M x K testing.
LOK, I could also say its hard to understand why you don't object to re-testing A in combinarion with B but do object to doing the same thing with NxMxK serialization library. But there are some differences. The serialization library is still one library with one person taking responsability for reviewing the results and reconciling any issues. any failure in any combination will be addressed. The serialization libary is change only once every few weeks whereas something in the whole of boost changed at least once / day. So the impact on testing time is much greater from retesting every AxB combination. Libraries A and B expect to change their public API only very occasionally. If the API is being tested independently, then re-testing by other libraries should be redundant. This doesn't apply to the internal behavior of any particular library. Regarding the serialization library in particular, I have strived to make the different aspects indepent and interact with each other only through the most narrow of API. This is the basic motivation for splitting the view of things in to archive and serialization. Similarly I strived to make the DLL versions identical in usage to the static library usage and all the archives implement the same API. The reason that we have the MxNxKxL(don't forget deug and release !) is that I was successful in doing this. Never-the-less, it has been a struggle and now the test results show pretty good othogonality except for compiler quirks which show some combinations trip compilers ICE's. I don't know that I could have arrived at this point without testing all the combinations from time to time. Libraries A & B don't have this situation. Robert Ramey