"Robert Ramey" <ramey@rrsd.com> writes:
As for not running the cross product of the various combinations - the problem is that failures now occur in just a couple of select situations that can't be predicted in advance. BCB failed on text archives only in some cases. STLPort 5.0 failed when using mapped collections. etc. Paring down the test suite would likely have missed these failures. The selected tests would all pass - thereby giving the false impression that the library was bug free.
Oh, there's a lot you can do to reduce it without losing any interesting data. For example, there is no reason to believe you're going to uncover serialization library bugs by testing against both static and dynamic runtimes, or especially by testing both singlethreaded and multithreaded builds, since you're not doing any threading in your tests. If you discover a test failure that appears only in one of these cases it is undoubtedly due to a compiler or standard library bug, not anything having to do with your code. Also, you could cut the testing time by 2/3 instantly if you'd just test all the archive types in a single executable, rather than building a separate executable for each one. And that would cover the case where multiple archives are in use, which I don't think you're covering today. -- Dave Abrahams Boost Consulting www.boost-consulting.com