[boost.test] feature request - random order of testing
Would it be possible to add ability to execute tests in random order, instead of as they are added into suite? Example: test_suite* init_unit_test_suite(int argc, char * argv[]) { test_suite* tests = BOOST_TEST_SUITE("..."); tests->add(BOOST_TEST_CASE(&test1)); tests->add(BOOST_TEST_CASE(&test2)); return tests; } Here test1() will be always executed before test2(). This behaviour may hide errors. It would be nice to have: - possibility to say: now execute test in random order, - if error happens framework would print out seed for random, - ability set seed manualy is needed to be able to repeat problem. (If such feature is already present in library my weak reading skills didn't spot it.) /Pavel
Would it be possible to add ability to execute tests in random order, instead of as they are added into suite?
Example:
test_suite* init_unit_test_suite(int argc, char * argv[]) { test_suite* tests = BOOST_TEST_SUITE("..."); tests->add(BOOST_TEST_CASE(&test1)); tests->add(BOOST_TEST_CASE(&test2)); return tests; }
Here test1() will be always executed before test2(). This behaviour may hide errors.
As a rule you shouldn't rely on order of test cases. Why not write 2 assertions that tests both orders?
It would be nice to have: - possibility to say: now execute test in random order, - if error happens framework would print out seed for random, - ability set seed manually is needed to be able to repeat problem.
I am plan in this release add an ability to run a test case by name. It may help you a bit. As for random order I will think about it, but it fills a bit incorrect (see above). Gennadiy.
"Gennadiy Rozental" <gennadiy.rozental@thomson.com> writes:
I plan in this release to add an ability to run a test case by name. It may help you a bit. As for random order I will think about it, but it fills a
Sorry. Meant *feels*.
bit incorrect (see above).
I don't know why; randomization of tests is good, as long as the seed is written when a test fails and a specific seed can be injected to reproduce the case. It means that your tests aren't unintentionally dependent on some arbitrary order or input data/size you may have chosen. I've used this strategy very successfully in the past. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
"Gennadiy Rozental" <gennadiy.rozental@thomson.com> wrote:
Would it be possible to add ability to execute tests in random order, ... As a rule you shouldn't rely on order of test cases. Why not write 2 assertions that tests both orders?
There may be many, many tests, each one specialized for certain functionality. /Pavel
participants (3)
-
David Abrahams -
Gennadiy Rozental -
Pavel Vozenilek