Re: [boost] Re: [Boost.Test] New testing procedure?
Jeff Garland wrote:
And that's today. Consider during the next couple months 3-4 new libraries are pending to be added. Serialization tests alone dramatically increase the length of the time to run the regression if we always run the full test. What will happen in a year when we have say 10 new libraries?
Robert and I have believe something will need to be done. We've tried to start a discussion, but no one responded:
http://lists.boost.org/MailArchives/boost/msg64471.php http://lists.boost.org/MailArchives/boost/msg64491.php
I don't know how feasable this would be, but you might also want to consider library dependancies so that if you modify the iterator adaptors library you can pull a list of tests that need to be re-run because the associated libraries use iterator adaptors. This would reduce the number of tests that need to be performed on each cycle, but you'd need some way of archiving the results and only updating the ones that are re-run. When testing a new platform/compiler/configuration, you can tell the test suite to run all the tests for that setup, but for the others (where it has already been run) it will operate as above, unless an explicit request for the full set of tests is provided (allowing for, say, the full set to be performed weekly while the dependant tests are performed daily/every 12 hours). I like the idea of specifying the test level (basic, torture, concept). Regards, Reece _________________________________________________________________ Want to block unwanted pop-ups? Download the free MSN Toolbar now! http://toolbar.msn.co.uk/
"Reece Dunn" <msclrhd@hotmail.com> writes:
Jeff Garland wrote:
And that's today. Consider during the next couple months 3-4 new libraries are pending to be added. Serialization tests alone dramatically increase the length of the time to run the regression if we always run the full test. What will happen in a year when we have say 10 new libraries?
Robert and I have believe something will need to be done. We've tried to start a discussion, but no one responded:
http://lists.boost.org/MailArchives/boost/msg64471.php http://lists.boost.org/MailArchives/boost/msg64491.php
I don't know how feasable this would be, but you might also want to consider library dependancies so that if you modify the iterator adaptors library you can pull a list of tests that need to be re-run because the associated libraries use iterator adaptors. This would reduce the number of tests that need to be performed on each cycle, but you'd need some way of archiving the results and only updating the ones that are re-run.
It's much, *much* simpler to just change the test system back to not re-run failing tests whose dependencies haven't changed. Once we've made that change, most test cycles will have nothing to do, and the rest will only test the stuff that might've changed.
When testing a new platform/compiler/configuration, you can tell the test suite to run all the tests for that setup, but for the others (where it has already been run) it will operate as above, unless an explicit request for the full set of tests is provided (allowing for, say, the full set to be performed weekly while the dependant tests are performed daily/every 12 hours).
I like the idea of specifying the test level (basic, torture, concept).
I am opposed to the idea of requiring humans to initiate the right tests, at least without proof that mechanically-initiated tests are unworkable. I don't think we've proven that yes. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
On Sat, 22 May 2004 13:18:41 -0400, David Abrahams wrote
I like the idea of specifying the test level (basic, torture, concept).
I am opposed to the idea of requiring humans to initiate the right tests, at least without proof that mechanically-initiated tests are unworkable. I don't think we've proven that yes.
They shouldn't be human initiated -- just configured to one level or another. Now obviously a human would need to intervene to reset the level, but that would only be in the case of ramping up to release, adding a new library, adding a new compiler, or a specific author request. I believe this would be rather infrequent. Basically, right now we have no way of setting up a Jamfile that can satisfy the need to have full up tests for new compilers, etc and a basic set of quick tests...short of replacing the Jamfile all the time. Jeff
"Jeff Garland" <jeff@crystalclearsoftware.com> writes:
On Sat, 22 May 2004 13:18:41 -0400, David Abrahams wrote
I like the idea of specifying the test level (basic, torture, concept).
I am opposed to the idea of requiring humans to initiate the right tests, at least without proof that mechanically-initiated tests are unworkable. I don't think we've proven that yes.
They shouldn't be human initiated -- just configured to one level or another. Now obviously a human would need to intervene to reset the level, but that would only be in the case of ramping up to release, adding a new library, adding a new compiler, or a specific author request. I believe this would be rather infrequent.
Basically, right now we have no way of setting up a Jamfile that can satisfy the need to have full up tests for new compilers, etc and a basic set of quick tests...short of replacing the Jamfile all the time.
I don't see that as a need, or neccessarily an advantage. Only outdated tests get run. That said, of course we do have a way: # foo.jam if $(Not_Defined) { # torture tests go here } In what way is commenting out the "if" line and checking in the jamfile significantly different from other mechanisms one might use to alter the set of tests that get run? -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
On Sat, 22 May 2004 17:12:26 -0400, David Abrahams wrote
"Jeff Garland" <jeff@crystalclearsoftware.com> writes: I don't see that as a need, or neccessarily an advantage. Only outdated tests get run.
You lost me on the outdated tests...
That said, of course we do have a way:
# foo.jam
if $(Not_Defined) { # torture tests go here }
In what way is commenting out the "if" line and checking in the jamfile significantly different from other mechanisms one might use to alter the set of tests that get run?
Because I don't have the capability to do that across all boost libraries. I'm thinking I should be able to say things like: (not real syntax) bjam --torture_test --static_link_only bjam --basic_test --dynamic_link_only bjam --torture_test --all_link_options I'm sure there's a way to set something like this up with Jamfile as well. Obviously for it to be useful boost-wide we would need to get library authors to agree to some standard. As for regression testers their automated setup could pick the level and linking options. The linking options would save time by not building 4 variations of a linked library for each compiler. The basic test level would save time by running less tests. Jeff
"Jeff Garland" <jeff@crystalclearsoftware.com> writes:
Because I don't have the capability to do that across all boost libraries. I'm thinking I should be able to say things like: (not real syntax) bjam --torture_test --static_link_only bjam --basic_test --dynamic_link_only bjam --torture_test --all_link_options
And we're going to have test servers all over the world doing different combinations of these? I just don't see how we'd manage it, and how the results would be really useful. And why should any server pick one particular kind of linking? What about the libraries that require a particular kind of linking? Is this runtime library linking or boost library linking selection?
I'm sure there's a way to set something like this up with Jamfile as well. Obviously for it to be useful boost-wide we would need to get library authors to agree to some standard. As for regression testers their automated setup could pick the level and linking options. The linking options would save time by not building 4 variations of a linked library for each compiler.
They don't do that now, unless the particular tests are configured to use all the different variations. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
On Sat, 22 May 2004 16:47:42 +0100, Reece Dunn wrote
I don't know how feasable this would be, but you might also want to consider library dependancies so that if you modify the iterator adaptors library you can pull a list of tests that need to be re-run because the associated libraries use iterator adaptors. This would reduce the number of tests that need to be performed on each cycle, but you'd need some way of archiving the results and only updating the ones that are re-run.
I think this is already how it is done, but I'm not 100% certain. Basically a cvs update and then running bjam should only rebuild and run the changed parts. The problem is, one update in the config and everthing is likely to recompile and rerun.
When testing a new platform/compiler/configuration, you can tell the test suite to run all the tests for that setup, but for the others (where it has already been run) it will operate as above, unless an explicit request for the full set of tests is provided (allowing for, say, the full set to be performed weekly while the dependant tests are performed daily/every 12 hours).
Yes, again I think that is already the case. But also, don't forget that we aren't really exercising all the variations with compiled libraries: static linking, dynamic linking, dyanmic multi-threaded, static multi-threaded. Honestly I'm still very suprised that with some compilers the exact same test that works with static linking fails with dynamic linking -- but it happens.
I like the idea of specifying the test level (basic, torture, concept).
Yeah, I think if we could agree on some standard levels that would give us a nice dimension of flexibility to run a quick test. Then it would be up to library developers to select some core subset for the basic tests. Jeff
participants (3)
-
David Abrahams -
Jeff Garland -
Reece Dunn