Robert Ramey wrote:
Basically you're correct on all of this.
Rene Rivera wrote:
After having run two cycles of tests for Boost with the mingw-3_4_2-stlport-5_0 configuration and having it take more than 14 hours on a 2.2Ghz+1GB machine, most of that in the Boost.Serialization library[*]. And reading some of the recent discussion about the desire to expand testing to include cross-version compatibility and cross-compiler compatibility, and hence having the number of tests multiply possibly exponentially. I am seriously concerned that we are going in the wrong direction when it comes to structuring tests.
This was the basis of my suggestion that we run a complete set only very occasionally.
I thought that was given. Sorry if I didn't make that clear when suggesting widening the current strategy. Clearly this isn't for everybody. [snip]
So there are a number of things that might be looked into
a) Reduce the combinations of the serializaton tests. b) Don't use libraries to test other libraries. That is don't re-test one library (.e.g. serialization) just because some other library that it depends upon (e.g. mpl) has changed. c) Define a two separate test Jamfiles - i) normal test ii) carpet bombing mode e) Maybe normal mode can be altered on frequent basis when I just want to test a new feature. or just one test. f) Include as part of the installation instructions for users an exhaustive test mode. That is a user how downloads and installs the package would have the option of producing the whole test results on his own platform and sending in his results. This would have a couple of advantages i) It would be sure that all new platforms are tested ii) I would ensure that the user has everyting installed correctly
a) I haven't thought about yet. I agree that the scheme could be better thought out, but it looks tricky to do it right, and my big issue right now, for a client, is to be able to test platform-independent archives (from different architectures) against each other in carpet-bomb mode. Since it turns out to be trivial to also accommodate testing different boost versions and compilers, I'm offering it up. b) is out of my hands. c) i) I have refactored the "normal tests" in places to support the existence of a carpet bomb mode, without lengthening the current "normal mode" tests. I've also been switching tests over to the autoregistering unit test framework as I go. c) ii) I have been working hard on. Carpet bombing is controlled by an environment variable (which I am considering changing to BOOST_SERIALIZATION_CARPET_BOMB). The tarball I posted recently is now very out-of-date, I have a lot of changes and have noticed a few additional things. For instance one needs a second class A that can be serialized by platform-independent archives, and the build environment will have to do this switching. Some primitives in the existent class A need to be removed in the platform independent case, and the use of std::rand() has to go, as you won't get the same stream of numbers on all platforms, so you use boost::random RNGs, and you need to be sure that you (re)seed the RNGs at appropriate times. d) If there's a carpet bombing mode, what do you call normal mode? Covering Fire? e) Sounds convenient. f) Whatever I come up with for c) ii) should be pretty easy to package like that, if you guys actually want to use it. -t