"Robert Ramey" <ramey@rrsd.com> writes:
David Abrahams wrote:
"Robert Ramey" <ramey@rrsd.com> writes:
As for not running the cross product of the various combinations - the problem is that failures now occur in just a couple of select situations that can't be predicted in advance. BCB failed on text archives only in some cases. STLPort 5.0 failed when using mapped collections. etc. Paring down the test suite would likely have missed these failures. The selected tests would all pass - thereby giving the false impression that the library was bug free.
Oh, there's a lot you can do to reduce it without losing any interesting data. For example, there is no reason to believe you're going to uncover serialization library bugs by testing against both static and dynamic runtimes,
I am testing for both static and dynamic runtimes.
I think you meant "not testing," which I see is true. You're testing against static and dynamic versions of your library.
When creating the DLL versions and also auto-linking it was different.
What was different?
It is A LOT more than just recompiling.
What is?
Now that things are working this is not necessary for debugging the serialization library itself.
or especially by testing both singlethreaded and multithreaded builds, since you're not doing any threading in your tests.
I don't believe I'm doing this. Currently, tests are built with the default threading model. That is, the Jamfile doesn't specify anything in this regard.
OK, maybe in my shock I overestimated. Sorry.
If you discover a test failure that appears only in one of these cases it is undoubtedly due to a compiler or standard library bug, not anything having to do with your code.
That was my original point. If I'm not making any changes, which I haven't been, almost no failures have anything to do with the serialization code.
But we want to find out when changes to other Boost libraries break it. And we want to find out when other people break it.
Also, you could cut the testing time by 2/3 instantly if you'd just test all the archive types in a single executable, rather than building a separate executable for each one.
Hmmm - that was what I did originally but it was objected to in the first review.
On what grounds?
Also, more compilers failed with due to resource exhaustion.
On some weaker compilers you might have to break a few tests into multiple translation units.
The current system has the benefit of permitting any user to test his own archive with all the tests. The documentation encourages users to do this with their own archives and I know they are doing it. I know this because when ever a user has a problem with his archive - I just tell him to run the whole test suite on it. The current system also lets me run just one archive by adding a command line argument - extremely useful.
I noticed. It's not only useful, but necessary, to cut down on the number of targets the Jamfile generates. All of that should be possible even with the tests built into a single executable.
I actually didn't have any idea how compilation and execution time varies by making the separation. I had just presumed that compilation was more or less proportional to the number of archives - given that a lot of time is spent on serialization code instantiation which would be proportional to the number of archives. Other code is already compiled in the library. Truth is since I run while I'm not watching I don't know how the time would vary. How do you know this?
I don't know for sure, but it's easy to see that every time you start a new compilation command there is overhead that can be avoided. Every time you include the same file in a new TU there's overhead that can be avoided. One eventually develops a pretty good instinct for these things after thinking about them a while.
And that would cover the case where multiple archives are in use, which I don't think you're covering today.
see test_mult_archive_types
In anycase, I have no problem with diminishing the the tests by just commenting out any combination of tests. Also the dynamic library versions can easily be suppressed by tweaking the Jamfile. As I said before, given that I'm making very few changes and I test those here, the boost test results are not used by me - or anyone else as far as I know.
The importance, to Boost as a whole, of having continuous testing of all libraries should not be underestimated. -- Dave Abrahams Boost Consulting www.boost-consulting.com