David Abrahams wrote:
"Robert Ramey" <ramey@rrsd.com> writes:
As for not running the cross product of the various combinations - the problem is that failures now occur in just a couple of select situations that can't be predicted in advance. BCB failed on text archives only in some cases. STLPort 5.0 failed when using mapped collections. etc. Paring down the test suite would likely have missed these failures. The selected tests would all pass - thereby giving the false impression that the library was bug free.
Oh, there's a lot you can do to reduce it without losing any interesting data. For example, there is no reason to believe you're going to uncover serialization library bugs by testing against both static and dynamic runtimes,
I am testing for both static and dynamic runtimes.When creating the DLL versions and also auto-linking it was different. It is A LOT more than just recompiling. Now that things are working this is not necessary for debugging the serialization library itself.
or especially by testing both singlethreaded and multithreaded builds, since you're not doing any threading in your tests.
I don't believe I'm doing this. Currently, tests are built with the default threading model. That is, the Jamfile doesn't specify anything in this regard.
If you discover a test failure that appears only in one of these cases it is undoubtedly due to a compiler or standard library bug, not anything having to do with your code.
That was my original point. If I'm not making any changes, which I haven't been, almost no failures have anything to do with the serialization code.
Also, you could cut the testing time by 2/3 instantly if you'd just test all the archive types in a single executable, rather than building a separate executable for each one.
Hmmm - that was what I did originally but it was objected to in the first review. Also, more compilers failed with due to resource exhaustion. The current system has the benefit of permitting any user to test his own archive with all the tests. The documentation encourages users to do this with their own archives and I know they are doing it. I know this because when ever a user has a problem with his archive - I just tell him to run the whole test suite on it. The current system also lets me run just one archive by adding a command line argument - extremely useful. I actually didn't have any idea how compilation and execution time varies by making the separation. I had just presumed that compilation was more or less proportional to the number of archives - given that a lot of time is spent on serialization code instantiation which would be proportional to the number of archives. Other code is already compiled in the library. Truth is since I run while I'm not watching I don't know how the time would vary. How do you know this?
And that would cover the case where multiple archives are in use, which I don't think you're covering today.
see test_mult_archive_types In anycase, I have no problem with diminishing the the tests by just commenting out any combination of tests. Also the dynamic library versions can easily be suppressed by tweaking the Jamfile. As I said before, given that I'm making very few changes and I test those here, the boost test results are not used by me - or anyone else as far as I know. Robert Ramey