Regression testing -- some new testing options?

Robert Ramey & I have been having a discussion offline about the impact of new boost libraries on the time it takes to run the regression tests. Serialization has a particularly time consuming test -- the so called 'torture test' suite. The problem is, that once serialization is in the tree it will be important to run that full test from time to time (like right before release), but it would probably be overkill to run everyday. And from past dicussion it is clear that running the regression test is already becoming very time consuming. Of course this isn't a serialization specific issue. With more and more libraries added the time to do any sort of regression test is continuing to grow. In addition, there is now the 'dll' versus 'static' library dimension for those of us with compiled libraries. It might be nice to sometimes run one or the other to speed things up. Also, Robert would like to be able to run 'debug' versus 'release' tests from time to time. So here's a suggestion. Perhaps it be possible to create a standard way we could set up the regression test targets so that library authors could: 1) define a couple standard test levels - say 'basic' and 'torture' 2) DLL versus static tests would be an option on the regression suite -- so you could run only one or both. Of course this isn't an issue for header libraries. 3) debug versus release is an option 4) define specific compiler/platform exclusions I believe such a design could increase the regression test effectiveness by allowing some contributors and users to run only one set of options (eg: 'basic-dll-release') tests. This option would be much-much faster than 'torture-all-all' -- which for a compiled library needs to compile and excute 4 versions of the tests (dll-release, dll-debug, static-release, static-debug). The compiler/platform exclusions would save time by just skipping over stuff that is going to fail and will never be fixed. There may be some way of doing this now, but ideally we would have some way of specifying this in the bjam file for the tests. Thoughts? Jeff
participants (1)
-
Jeff Garland