boost::test: expected error leads to failure - why?
Hi! When using the boost::test unit-test framework calls like boost::unit_test::test_suite* test = BOOST_TEST_SUITE( "Testing Something" ); test->add(BOOST_TEST_CASE(&test_function_with_expected_error), 1); lead to an error in return code when used together with VC7 as a post-build step Of course I could use runtime argument "--result_code=no", but then how can I make a difference between expected and unexpected errors? It makes sense to me to have expected errors in a testsuite, and especially to declare a test failure when an expected error does not appear. Could you please explain how boost::test supports this without having to write "failure-tests" myself? Markus
Hi!
When using the boost::test unit-test framework calls like
boost::unit_test::test_suite* test = BOOST_TEST_SUITE( "Testing Something" );
test->add(BOOST_TEST_CASE(&test_function_with_expected_error), 1);
lead to an error in return code when used together with VC7 as a post-build step
It shouldn't. Could you provide an example to replicate an error?
Of course I could use runtime argument "--result_code=no", but then how can I make a difference between expected and unexpected errors?
Check "Usage recomendation" section in docs. Use above flag always for post-build event
It makes sense to me to have expected errors in a testsuite, and especially to declare a test failure when an expected error does not appear. Could you please explain how boost::test supports this without having to write "failure-tests" myself?
Markus
Gennadiy
Gennadiy Rozental
Hi!
When using the boost::test unit-test framework calls like
boost::unit_test::test_suite* test = BOOST_TEST_SUITE( "Testing Something" );
test->add(BOOST_TEST_CASE(&test_function_with_expected_error), 1);
lead to an error in return code when used together with VC7 as a post-build step
It shouldn't. Could you provide an example to replicate an error?
I extended a unit_test example file and built a vc7 project. No more gimmicks. Should be reproducable with any of the example files, since the expected error came from stuff like void free_test_function() { BOOST_CHECK( 2 == 1 ); }
Of course I could use runtime argument "--result_code=no", but then how can I make a difference between expected and unexpected errors?
Check "Usage recomendation" section in docs. Use above flag always for post-build event
This makes no sense to me (yet). Sorry if I ask again: Why should I use this flag? I really want my build to _have_ a failure, when the testsuite has an _unexpected_ failure, so I really need --result-code not to be 'no'. OTOH I want my build to succeed in the vicinity of an expected failure, regardless of the value of --result-code. This was the aim of my OP. Yet unsure if I understood the architecture of boost::test ... Markus
"Markus Werle"
Gennadiy Rozental
writes: Hi!
When using the boost::test unit-test framework calls like
boost::unit_test::test_suite* test = BOOST_TEST_SUITE( "Testing Something" );
test->add(BOOST_TEST_CASE(&test_function_with_expected_error), 1);
lead to an error in return code when used together with VC7 as a post-build step
It shouldn't. Could you provide an example to replicate an error?
I extended a unit_test example file and built a vc7 project. No more gimmicks. Should be reproducable with any of the example files, since the expected error came from stuff like
void free_test_function() { BOOST_CHECK( 2 == 1 ); }
I couldn't replicate any problems. An error code is zero if you specified expected exception.
Of course I could use runtime argument "--result_code=no", but then how can I make a difference between expected and unexpected errors?
Check "Usage recomendation" section in docs. Use above flag always for post-build event
This makes no sense to me (yet). Sorry if I ask again: Why should I use this flag? I really want my build to _have_ a failure, when the testsuite has an _unexpected_ failure, so I really need --result-code not to be 'no'.
It will. VC gui does not rely solely on a return code to report a build failure. It parse a build output for error statements that weill be present. If you did not specify an result_code=no you will get an extra error that you are not interrested in.
OTOH I want my build to succeed in the vicinity of an expected failure, regardless of the value of --result-code. This was the aim of my OP.
See above. An error is going to be reported regardless. Gennadiy
Gennadiy Rozental
It will. VC gui does not rely solely on a return code to report a build failure. It parse a build output for error statements that weill be present.
This could be a hint for the unwanted behaviour. I'll test that next week and give you more details.
If you did not specify an result_code=no you will get an extra error that you are not interrested in.
Sorry, I am too slow to follow here. Assuming the output contains nothing for vc7 to parse that triggers an error, the result code is (should be) 0 even in the vicinity of expected failures - right? Again: why must I switch off the result code for vc7 in such a case? Markus
If you did not specify an result_code=no you will get an extra error that you are not interested in.
Sorry, I am too slow to follow here. Assuming the output contains nothing for vc7 to parse that triggers an error, the result code is (should be) 0 even in the vicinity of expected failures - right?
Again: why must I switch off the result code for vc7 in such a case?
In case of success there is no difference. In both cases return code is zero. In case of failure though is's better for GUI to ignore error result code from test. The errors are repotyed anyway.
Markus
Gennadiy
participants (2)
-
Gennadiy Rozental -
Markus Werle