[test] BOOST_TEST - universal testing tool
Hi, Based on a discussion on mini-review thread I make some changes to the universal testing tool interface and now it look as follows (similar interfaces on WARN and REQUIRE levels): BOOST_CHECK_ASSERTION(A) - new expression template based testing tool BOOST_TEST(A,M) := BOOST_CHECK_ASSERTION(A) if M is not supplied or BOOST_CHECK_MESSAGE(A,M) if M is supplied. BOOST_TEST (and it's siblings BOOST_TEST_WARN and BOOST_TEST_REQUIRE) is intended to become a primary (only?) testing tools from now on. It replaces and deprecates right away following tools: BOOST_CHECK BOOST_CHECK_MESSAGE BOOST_CHECK_EQUAL, BOOST_CHECK_NE, BOOST_CHECK_LT, BOOST_CHECK_LE, BOOST_CHECK_GT, BOOST_CHECK_GE There are still some tools which are not directly replaceable. I have some ideas how we can get there, but your input is appreciated: 1. BOOST_CHECK_THROW, BOOST_CHECK_EXCEPTION, BOOST_CHECK_NO_THROW We can probably leave these as is. Alternatively we'll need to postpone expression evaluation with separate BOOST_TEST_DELAYED tool. We also need to introduce special tags unit_test::throws, unit_test::no_throw: BOOST_TEST_DELAYED(foo(), unit_test::throws<my_exception>() ); BOOST_TEST_DELAYED(foo(), unit_test::throws<my_exception>(predicate) ); BOOST_TEST_DELAYED(foo(), unit_test::throws( my_exception(...) ) ); BOOST_TEST_DELAYED(foo(), unit_test::no_throw() ); Later form will match to exact exception value. We might be able to fit into BOOST_TEST interface, but that would require somehow we need to recognize executable entity as special case: BOOST_TEST( foo, unit_test::throws<my_exception>() ); Not quite sure if this is possible. 2. BOOST_CHECK_CLOSE, BOOST_CHECK_CLOSE_FRACTION, BOOST_CHECK_SMALL We can deal with these by introducing couple special tags: BOOST_TEST(a == 1.5, unit_test::tolerance(1e-6) ); BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) ); BOOST_TEST(a, unit_test::small(1e-3) ); On a plus side we'll be able to use these in new previously unavailable way: BOOST_TEST(a > 1.1, unit_test::tolerance(1e-6) ); This would test that value of a either > 1.1 or within a fraction tolerance of it. 3. BOOST_CHECK_PREDICATE Similar to (1) we need to delay predicate execution. Might be easier to leave it be. 4. BOOST_CHECK_EQUAL_COLLECTIONS I think using overloading of equality comparisons we should be able to support this directly: BOOST_TEST( expr == std::vector<int>{1,2,3,4}); 5. BOOST_CHECK_BITWISE_EQUAL We can introduce yet another tag to deal with this: BOOST_TEST(a == 0xA1FB, unit_test::bitwise ); As usual any comments are welcome. Gennadiy
Gennadiy Rozental <rogeeff@gmail.com> writes:
Hi,
Based on a discussion on mini-review thread I make some changes to the universal testing tool interface and now it look as follows (similar interfaces on WARN and REQUIRE levels):
BOOST_CHECK_ASSERTION(A) - new expression template based testing tool
BOOST_TEST(A,M) := BOOST_CHECK_ASSERTION(A) if M is not supplied or BOOST_CHECK_MESSAGE(A,M) if M is supplied.
BOOST_TEST (and it's siblings BOOST_TEST_WARN and BOOST_TEST_REQUIRE) is intended to become a primary (only?) testing tools from now on. It replaces and deprecates right away following tools:
BOOST_CHECK BOOST_CHECK_MESSAGE BOOST_CHECK_EQUAL, BOOST_CHECK_NE, BOOST_CHECK_LT, BOOST_CHECK_LE, BOOST_CHECK_GT, BOOST_CHECK_GE
This should simplify test code a lot. But, as it's now the default tool, has any progress been made on using it with C++03 compilers? Alex
Alexander Lamaison <awl03 <at> doc.ic.ac.uk> writes:
This should simplify test code a lot. But, as it's now the default tool, has any progress been made on using it with C++03 compilers?
It actually works for quite some number of cases. You can see in test_tools_test unit test in here: http://beta.boost.org/development/tests/trunk/developer/test.html. There are some use cases which are not supported (you can see them ifdefed out). Gennadiy
Le 05/11/12 08:16, Gennadiy Rozental a écrit :
Alexander Lamaison <awl03 <at> doc.ic.ac.uk> writes:
This should simplify test code a lot. But, as it's now the default tool, has any progress been made on using it with C++03 compilers? It actually works for quite some number of cases. You can see in test_tools_test unit test in here:
http://beta.boost.org/development/tests/trunk/developer/test.html.
There are some use cases which are not supported (you can see them ifdefed out).
Gennadiy
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Hi, there is a regression with some compilers ( intel- darwin- 11.1, clang- darwin- trunk, clang- darwin- trunk c++11, gcc- 4.7.1_0x), all of them having the same symptoms Run [2012-11-05 13:48:26 UTC]: fail terminate called after throwing an instance of '__gnu_cxx::recursive_init_error' what(): std::exception All the test using Boost.Test are failing. Please, could you take care of this issue? Best, Vicente
Hi Gennadiy, The new interface looks very attractive. Below I enclose two suggestions that are orthogonal (I think) to your changes, but I thought it was a good time to suggest them. 2012/11/4 Gennadiy Rozental <rogeeff@gmail.com>
2. BOOST_CHECK_CLOSE, BOOST_CHECK_CLOSE_FRACTION, BOOST_CHECK_SMALL
We can deal with these by introducing couple special tags:
BOOST_TEST(a == 1.5, unit_test::tolerance(1e-6) ); BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) ); BOOST_TEST(a, unit_test::small(1e-3) );
On a plus side we'll be able to use these in new previously unavailable way: BOOST_TEST(a > 1.1, unit_test::tolerance(1e-6) );
This would test that value of a either > 1.1 or within a fraction tolerance of it.
In my job, we are using Boost.Test in a commercial app. We are doing (and testing) some basic computations on physical quantities. It turned out that for around 95% of the checks on floating-point values we need the very same (percent) tolerance. And it is inconvenient to have to type the same constant time and again. It would be nice if there was a way to define (as an option) a "default tolerance". With the solution you propose above, having to type this tag "unit_test::percent_tolerance" would incur even more "syntactic noise" in cases where the tolerance is "obvious" to the reader. Here, the concept of default tolerance would be even more useful. Although I am not sure if this is doable. But perhaps it is: if I am able to configure the framework so that when it sees a check on floating-point numbers it assumes we check with tolerance: default one. What do you think? 4. BOOST_CHECK_EQUAL_COLLECTIONS
I think using overloading of equality comparisons we should be able to support this directly:
BOOST_TEST( expr == std::vector<int>{1,2,3,4});
Will that work? I.e., in case of any two ranges, not necessarily containers? If there is some even more clever (than my idea) suggestion to do it it is even better. What I was missing from Boost.Test for quite a while was some additional check that would accept as input two ranges rather than four iterators. "Range" R in this sense is any object for which expressions begin(R) and end(R) are valid: a container, an array, a pair of iterators. I often wanted to compare a vector returned by value from one expression with an array of values, and could not do it without introducing unnecessary additional variables: I had to write: int reference[] = {1, 2, 4}; auto&& answer = FindAnswer(); BOOST_CHECK_EQUAL_COLLECTIONS(begin(reference), end(reference), begin(answer), end(answer)); I would like to write: int reference[] = {1, 2, 4}; BOOST_CHECK_EQUAL_COLLECTIONS(reference, FindAnswer()); Or (if this is doable): BOOST_CHECK_EQUAL_COLLECTIONS(FindAnswer(), {1, 2, 4}); Regards, &rzej
Andrzej Krzemienski wrote
Hi Gennadiy, The new interface looks very attractive. Below I enclose two suggestions that are orthogonal (I think) to your changes, but I thought it was a good time to suggest them.
2012/11/4 Gennadiy Rozental <
rogeeff@
>
2. BOOST_CHECK_CLOSE, BOOST_CHECK_CLOSE_FRACTION, BOOST_CHECK_SMALL
We can deal with these by introducing couple special tags:
BOOST_TEST(a == 1.5, unit_test::tolerance(1e-6) ); BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) ); BOOST_TEST(a, unit_test::small(1e-3) );
On a plus side we'll be able to use these in new previously unavailable way: BOOST_TEST(a > 1.1, unit_test::tolerance(1e-6) );
This would test that value of a either > 1.1 or within a fraction tolerance of it.
In my job, we are using Boost.Test in a commercial app. We are doing (and testing) some basic computations on physical quantities. It turned out that for around 95% of the checks on floating-point values we need the very same (percent) tolerance. And it is inconvenient to have to type the same constant time and again. It would be nice if there was a way to define (as an option) a "default tolerance". With the solution you propose above, having to type this tag "unit_test::percent_tolerance" would incur even more "syntactic noise" in cases where the tolerance is "obvious" to the reader. Here, the concept of default tolerance would be even more useful. Although I am not sure if this is doable. But perhaps it is: if I am able to configure the framework so that when it sees a check on floating-point numbers it assumes we check with tolerance: default one.
What do you think?
it seems easy for you to define your specific macro #define MY_FP_TEST(E) BOOST_TEST(E, my_tolerance ); and change the my_tolerance as you want. Vicente -- View this message in context: http://boost.2283326.n4.nabble.com/test-BOOST-TEST-universal-testing-tool-tp... Sent from the Boost - Dev mailing list archive at Nabble.com.
BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) );
It turned out that for around 95% of the checks on floating-point values we need the very same (percent) tolerance. And it is inconvenient to have to type the same constant time and again. It would be nice if there was a way to define (as an option) a "default tolerance".
Could you simply macro-up the default tolerance? #define MY_TEST_WITH_DEFAULT_TOLERANCE(expr) \ BOOST_TEST(expr, unit_test::percent_tolerance(whatever)) - Rhys
BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) );
...
BOOST_TEST( expr == std::vector<int>{1,2,3,4});
Can combine these constructs to perform floating point tests with tolerance on a collection? That is, BOOST_TEST( expr == std::vector<double>{1,2,3,4}, unit_test::percent_tolerance(1e-6)); I ask because, for my numerics codes, I've kludged together the Test floating point tolerance algorithms to produce something like a BOOST_CHECK_CLOSE_COLLECTIONS. Direct support within Test would be much appreciated. Thanks, Rhys
Rhys Ulerich <rhys.ulerich <at> gmail.com> writes:
BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) );
...
BOOST_TEST( expr == std::vector<int>{1,2,3,4});
Can combine these constructs to perform floating point tests with tolerance on a collection?
That is, BOOST_TEST( expr == std::vector<double>{1,2,3,4}, unit_test::percent_tolerance(1e-6));
Hmmm... Good point. I'll see if this is possible. As for "default tolerance", I am afraid it might be a bit surprising if plain comparison of doubles suddenly will start to use tolerance without user say so. Unless it is set to zero by default in which case we are back to usual semantic. Once you set it to something non zero we can probably switch to "close" comparison. What about the scope of the tolerance? Should it be global or per test case? Gennadiy
2012/11/6 Gennadiy Rozenal <rogeeff@gmail.com>
Rhys Ulerich <rhys.ulerich <at> gmail.com> writes:
BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) );
...
BOOST_TEST( expr == std::vector<int>{1,2,3,4});
Can combine these constructs to perform floating point tests with tolerance on a collection?
That is, BOOST_TEST( expr == std::vector<double>{1,2,3,4}, unit_test::percent_tolerance(1e-6));
Hmmm... Good point. I'll see if this is possible.
As for "default tolerance", I am afraid it might be a bit surprising if plain comparison of doubles suddenly will start to use tolerance without user say so. Unless it is set to zero by default in which case we are back to usual semantic. Once you set it to something non zero we can probably switch to "close" comparison.
What about the scope of the tolerance? Should it be global or per test case?
From the perspective of my usage, I guess it could be set per suite, if
this is possible. Per-test would require too much repetition, unless I could define it in my fixture and have the test framework somehow read it from the fixture. Regards, &rzej
Andrzej Krzemienski <akrzemi1 <at> gmail.com> writes:
From the perspective of my usage, I guess it could be set per suite, if this is possible. Per-test would require too much repetition, unless I could define it in my fixture and have the test framework somehow read it from the fixture.
What about interface like this: BOOST_TEST_DECORATORS( + decorator::tolerance(1e-6) ) BOOST_AUTO_TEST_CASE( foo ) { BOOST_TEST( foo == 1.5 ); } BOOST_TEST_DECORATORS( + decorator::percent_tolerance(1e-5) ) BOOST_AUTO_TEST_SUITE(s1) BOOST_AUTO_TEST_CASE( foo ) { BOOST_TEST( foo == 1.5 ); } BOOST_AUTO_TEST_SUITE_END Similar functionality can be implemented using fixture decorator, which already exists. Gennadiy
2012/11/6 Gennadiy Rozental <rogeeff@gmail.com>
Andrzej Krzemienski <akrzemi1 <at> gmail.com> writes:
From the perspective of my usage, I guess it could be set per suite, if this is possible. Per-test would require too much repetition, unless I could define it in my fixture and have the test framework somehow read it from the fixture.
What about interface like this:
BOOST_TEST_DECORATORS( + decorator::tolerance(1e-6) ) BOOST_AUTO_TEST_CASE( foo ) { BOOST_TEST( foo == 1.5 ); }
BOOST_TEST_DECORATORS( + decorator::percent_tolerance(1e-5) ) BOOST_AUTO_TEST_SUITE(s1)
BOOST_AUTO_TEST_CASE( foo ) { BOOST_TEST( foo == 1.5 ); }
BOOST_AUTO_TEST_SUITE_END
Similar functionality can be implemented using fixture decorator, which already exists.
This is nice. How does this work if I spread my suite across multiple source files? do I need to repeat the decorators in each file? Regards, &rzej
Andrzej Krzemienski <akrzemi1 <at> gmail.com> writes:
This is nice. How does this work if I spread my suite across multiple source files? do I need to repeat the decorators in each file?
No. Decorators will stick to the test suite. Gennadiy
What about interface like this:
BOOST_TEST_DECORATORS( + decorator::tolerance(1e-6) ) BOOST_AUTO_TEST_CASE( foo ) { BOOST_TEST( foo == 1.5 ); }
This just feels like a fixture-- any reason a regular ol' per-suite or per-test fixture providing a "default_tolerance" member wouldn't suffice? - Rhys
Rhys Ulerich <rhys.ulerich <at> gmail.com> writes:
What about interface like this:
BOOST_TEST_DECORATORS( + decorator::tolerance(1e-6) ) BOOST_AUTO_TEST_CASE( foo ) { BOOST_TEST( foo == 1.5 ); }
This just feels like a fixture-- any reason a regular ol' per-suite or per-test fixture providing a "default_tolerance" member wouldn't suffice?
It is (and it can be implemented like one), but if this is common need why not make cleaner interface for it? Gennadiy
This just feels like a fixture-- any reason a regular ol' per-suite or per-test fixture providing a "default_tolerance" member wouldn't suffice?
It is (and it can be implemented like one), but if this is common need why not make cleaner interface for it?
Because expanding the Boost.Test API seems to cause users to ask for very, very specific features rather than to realize that the existing functionality supports vastly richer use cases than what most folks need. And because fixtures are a wonderful gateway drug. :) - Rhys
participants (7)
-
Alexander Lamaison -
Andrzej Krzemienski -
Gennadiy Rozenal -
Gennadiy Rozental
-
Rhys Ulerich -
Vicente Botet -
Vicente J. Botet Escriba