Hi, This update includes my first steps in new areas of interaction based testing. One important branch is automated exception safety testing. This scheme was originally used by Dave A. in his work for STLPort testing. And also there was some variation covered in April 2001 CUJ. To employ this new facility one need to include exception_safety.hpp and (most probable) mock_object.hpp. The interface is a single invocation function: namespace itest { void exception_safety( unit_test::callback0<> const& F, unit_test::const_string test_name = "" ); } The function under test should include some statements to be tested for exception safety and checks for invariants using regular Test Tools. Boost.Test will make sure all possible execution patches are tested Faking failures in different locations. Any Test Tool failure is treated as failed invariant and abort specific execution path. Interface also include a helper macro that support auto registration and slightly simplify users life: ------------------------------------------ BOOST_TEST_EXCEPTION_SAFETY( test_name ). ------------------------------------------ Here is an example usage: --------------- // Here is an exempt from simple (incorrect) stack implementation; It is instrumented to better identify failure locations template<class T> class stack { public: explicit stack( int init_capacity = 10 ) : m_capacity( init_capacity ) , m_size( 0 ) , m_v( BOOST_ITEST_NEW(T)[m_capacity] ) { BOOST_ITEST_SCOPE( stack::stack ); } ~stack() { delete[] m_v; } void push( T const& element ) { BOOST_ITEST_SCOPE( stack::push ); if( m_size == m_capacity ) { m_capacity *= 2; T* new_buffer = BOOST_ITEST_NEW( T )[m_capacity]; for( unsigned i = 0; i < m_size; i++ ) { new_buffer[i] = m_v[i]; } delete [] m_v; m_v = new_buffer; } m_v[m_size++] = element; } unsigned size() { return m_size; } private: unsigned m_capacity; unsigned m_size; T* m_v; }; //____________________________________________________________________________// BOOST_TEST_EXCEPTION_SAFETY( test_stack_push ) { stack<mock_object<> > st( 2 ); for( unsigned i = 0; i < 3; ++i ) { try { st.push( simple_mock ); } catch( ... ) { // this invariant checks that in case of failed push number of elements doesn't change BOOST_CHECK_EQUAL( i, st.size() ); throw; } } } This examples shows memory leaks and failed invariants in specific execution paths. Execution path is reported and leaking memory is dumped. New parameter is introduced for Unit Test framework: --break_exec_path=<test_name>:<execution_path_index> Using parameter in conjunction with --catch_system_error=no should allow you to see an execution path causing a memory leak in debugger(Boost.Test will force an abort in failure location). test_name above is the name of the test case and execution_path_index is reported index of execution path containing a memory leak. I would appreciate any comments and subjections for this new facility. Regards, Gennadiy
Gennadiy Rozental wrote:
This update includes my first steps in new areas of interaction based testing. One important branch is automated exception safety testing. This scheme was originally used by Dave A. in his work for STLPort testing. And also there was some variation covered in April 2001 CUJ.
Hi, I'm trying to change some of my tests to use this, but I'm having a few problems, so I've got a few requests. Sorry if you already do some of these. It would be good to able to use my own exception types instead of Boost.Test's private exception type. You could possibly write a version of BOOST_ITEST_EPOINT which lets the user specify their own exception type, and some kind of mechanism to specify which exceptions are allowed seems essential. It would also be very useful to be able to tell where an exception was thrown from. Since you're already tracking scopes, maybe there could be a way of telling if it was in a specific scope? My tests store some data in globals (in anonymous namespaces), and these are causing the memory leak detector to complain. So can I turn it off? I'm using a custom allocator which detect leaks anyway, so I don't need it. BOOST_CHECK now seems to throw an exception, but I've been using it in destructors. So is this really needed? If it is, can you make the exception type public so I can treat it differently to other exception types. Also, exception_safety.hpp should include <boost/bind.hpp>. Daniel
It would be good to able to use my own exception types instead of Boost.Test's private exception type. You could possibly write a version of BOOST_ITEST_EPOINT which lets the user specify their own exception type, and some kind of mechanism to specify which exceptions are allowed seems essential.
Why would you need an ability to specify which fake exception to throw? This facility tests namely this ability of your code to do unwinding properly in case of any exception in any place. You don't really need to know which exception it was. To check post exception invariants you could use catch(...)
It would also be very useful to be able to tell where an exception was thrown from. Since you're already tracking scopes, maybe there could be a way of telling if it was in a specific scope?
If an execution path causing a memory leaks and failed invariant log shows an execution path traceback (try est_example2). An exception is marked by "Failure point". Also you could specify --break_exec_path parameter and execution will about in failure point.
My tests store some data in globals (in anonymous namespaces), and these are causing the memory leak detector to complain. So can I turn it off?
1. Keep in mind that in general exception safety test meant for small local checks and not intended for huge code pieces. Accordingly all global data should be initialized outside of test scope (usually, unless you are doing the global init testing)
I'm using a custom allocator which detect leaks anyway, so I don't need it. If you are using custom allocator you shouldn't be affected by operator new overloads. In some cases you could implement class level new to prevent overloaded new invocation.
As a last resort there is a macro BOOST_ITEST_NO_NEW_OVERLOADS that needs to be defined both for library and test compilation. But then nothing is going to be caught by my facilities.
BOOST_CHECK now seems to throw an exception, but I've been using it in destructors. So is this really needed? If it is, can you make the exception type public so I can treat it differently to other exception types.
This test is not intended to perform anything else but exception safety test. So DONT use Test Tools inside your test code unless you are testing for invariants. Any Test Tools failure is treated as failed invariant and execution path is aborted. No need to catch this exception - error found already.
Also, exception_safety.hpp should include <boost/bind.hpp>.
I did not included it intentionally, since to use macroless interface you do not need one and I did not want to incur unnecessary dependencies. Gennadiy
Gennadiy Rozental wrote:
It would be good to able to use my own exception types instead of Boost.Test's private exception type. You could possibly write a version of BOOST_ITEST_EPOINT which lets the user specify their own exception type, and some kind of mechanism to specify which exceptions are allowed seems essential.
Why would you need an ability to specify which fake exception to throw? This facility tests namely this ability of your code to do unwinding properly in case of any exception in any place. You don't really need to know which exception it was. To check post exception invariants you could use catch(...)
Unordered associative containers have a few exception specifications like (from TR1 6.3.1.1/2): "For unordered associative containers, if an exception is thrown by any operation other than the container’s hash function from within an insert() function inserting a single element, the insert() function has no effect." So the tests need be to sensitive to where the exception was thrown. Currently I'm dealing this by throwing different types of exceptions from different methods. Another, possibly better, way of doing it is to be able to find out the location of the throw (maybe by using the scope). It would also be possible to have multiple tests using different objects to deal the different circumstances.
My tests store some data in globals (in anonymous namespaces), and these are causing the memory leak detector to complain. So can I turn it off?
1. Keep in mind that in general exception safety test meant for small local checks and not intended for huge code pieces. Accordingly all global data should be initialized outside of test scope (usually, unless you are doing the global init testing)
This seems quite limiting to me, but in my case I can work around it by defining a class level new as the false leaks aren't from the code being tested, but the tests. I'm getting reports of memory leaks as they only get cleaned up when the full test ends.
BOOST_CHECK now seems to throw an exception, but I've been using it in destructors. So is this really needed? If it is, can you make the exception type public so I can treat it differently to other exception types.
This test is not intended to perform anything else but exception safety test. So DONT use Test Tools inside your test code unless you are testing for invariants. Any Test Tools failure is treated as failed invariant and execution path is aborted. No need to catch this exception - error found already.
I have a couple of classes that I use for testing for strong and basic exception safety. In order to avoid writing try/catches everywhere they do their tests in the destructor. So this is an exception safety test, or at least an exception spefi
Also, exception_safety.hpp should include <boost/bind.hpp>.
I did not included it intentionally, since to use macroless interface you do not need one and I did not want to incur unnecessary dependencies.
Gennadiy
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Daniel James wrote:
I have a couple of classes that I use for testing for strong and basic exception safety. In order to avoid writing try/catches everywhere they do their tests in the destructor. So this is an exception safety test, or at least an exception spefi
I'm not sure what happened there. I think I might have hit undo a couple of times too many. It's meant to say: "or at least an exception specification test."
Why would you need an ability to specify which fake exception to throw? This facility tests namely this ability of your code to do unwinding properly in case of any exception in any place. You don't really need to know which exception it was. To check post exception invariants you could use catch(...)
Unordered associative containers have a few exception specifications like (from TR1 6.3.1.1/2):
"For unordered associative containers, if an exception is thrown by any operation other than the containers hash function from within an insert() function inserting a single element, the insert() function has no effect."
So the tests need be to sensitive to where the exception was thrown.
Why? I do not follow you. Test needs to check that if any function call other then hash function invocation throw *any* exception specific invariant (in this case: size() is the same) is satisfied.
Currently I'm dealing this by throwing different types of exceptions from different methods. Another, possibly better, way of doing it is to be able to find out the location of the throw (maybe by using the scope).
I am sorry this is completely unclear to me. Maybe some code example?
It would also be possible to have multiple tests using different objects to deal the different circumstances.
And here?
My tests store some data in globals (in anonymous namespaces), and these are causing the memory leak detector to complain. So can I turn it off?
1. Keep in mind that in general exception safety test meant for small local checks and not intended for huge code pieces. Accordingly all global data should be initialized outside of test scope (usually, unless you are doing the global init testing)
This seems quite limiting to me, but in my case I can work around it by defining a class level new as the false leaks aren't from the code being tested, but the tests. I'm getting reports of memory leaks as they only get cleaned up when the full test ends.
I realize that global operator new overload bound to cause some false positives under some circumstances. Users will need to address this in their code. If you have any better proposition I am all for it.
BOOST_CHECK now seems to throw an exception, but I've been using it in destructors. So is this really needed? If it is, can you make the exception type public so I can treat it differently to other exception types.
This test is not intended to perform anything else but exception safety test. So DONT use Test Tools inside your test code unless you are testing for invariants. Any Test Tools failure is treated as failed invariant and execution path is aborted. No need to catch this exception - error found already.
I have a couple of classes that I use for testing for strong and basic exception safety. In order to avoid writing try/catches everywhere they do their tests in the destructor.
To check basic ES you don't need to catch any exceptions. Framework will check that you do unwinding properly by making sure you are not leaking any resources. To check for invariants (strong ES) my recommendation is to use catch(...) and do this on test function level preferably. Every ES test consist of series of tests for all possible execution paths. Failed invariant abort the execution path testing and skip to next.
So this is an exception safety test, or at least an exception specification test
This is two completely different things. This facility only bothers with exception safety.
Gennadiy Rozental wrote:
To check basic ES you don't need to catch any exceptions. Framework will check that you do unwinding properly by making sure you are not leaking any resources.
That isn't exactly true. The class still needs to meet it's invariants. For example, if an exception is thrown while rehashing a hash table, it will probably be left with only some of its elements - so the test needs to check that the size has been changed to match the decreased number of elements. Of course, that's possible within your framework. I'll try to give you an idea about what I'm doing. The example I gave was that a single element insert has the strong guarantee unless an exception was thrown from the hash object. My current way of testing this is something like: unordered_set<test::mock_object, test::hash, test::equal, test::allocator> x; unordered_strong_test tester(x); try { x.insert(1); } catch(hash_exception) { throw; } catch(...) { tester.test(); throw; } 'strong_test' is a class which stores several details about x to be check that they haven't changed. An alternative way of doing this would be something like: unordered_strong_test tester(x); try { x.insert(1); } catch(...) { if(thrown_from_scope("hash")) tester.test(); throw; } Which I think I prefer as it's a bit more flexible. I think this is possible within your framework, although not really supported? Another version would be to have two separate tests, the first to test for basic safety: unordered_set<test::mock_object, test::hash, test::equal, test::allocator> x; x.insert(1); and one to test for strong safety, with a non-throwing hash: boost::unordered_set<test::mock_object, test::hash_nothrow, test::equal, test::allocator> x; unordered_strong_test tester(x); try { x.insert(1); } catch(...) { tester.test(); throw; } Incidently, my tests aren't actually written like this. unordered_strong_test works more like a scope guard, something like: unordered_strong_test tester(x); try { x.insert(1); tester.dismiss(); } catch(hash_exception) { tester.dismiss(); throw; } I also have a similar class which tests the class invariants - in order to test basic exception safety. It doesn't have a 'dismiss' method as I want to run the test regardless. This is why I would like to be able to use BOOST_CHECK in the destructor, but that's more of a convenience than anything. I hope that makes things clearer... Daniel
"Daniel James" <daniel@calamity.org.uk> wrote in message news:doa5fo$fd6$1@sea.gmane.org...
Gennadiy Rozental wrote:
To check basic ES you don't need to catch any exceptions. Framework will check that you do unwinding properly by making sure you are not leaking any resources.
That isn't exactly true. The class still needs to meet it's invariants.
No. This is not what a definition says.
For example, if an exception is thrown while rehashing a hash table, it will probably be left with only some of its elements - so the test needs to check that the size has been changed to match the decreased number of elements. Of course, that's possible within your framework.
This is requirement for strong guarantee
I'll try to give you an idea about what I'm doing. The example I gave
[...]
Another version would be to have two separate tests, the first to test for basic safety:
unordered_set<test::mock_object, test::hash, test::equal, test::allocator> x; x.insert(1);
and one to test for strong safety, with a non-throwing hash:
boost::unordered_set<test::mock_object, test::hash_nothrow, test::equal, test::allocator> x; unordered_strong_test tester(x); try { x.insert(1); } catch(...) { tester.test(); throw; }
This is a correct way of structuring your tests. Don't mix basic and strong test withing the same test case. TDD actually recommends to have separate test case for every single assertion. And in this design you do not need to know where exception was thrown from. It's possible that your class needs to check different invariants depending on where the failure occur. Do NOT do this within same test case. Use two - with different mocks throwing exceptions in different locations and different invariants checks.
Incidently, my tests aren't actually written like this. unordered_strong_test works more like a scope guard, something like:
unordered_strong_test tester(x); try { x.insert(1); tester.dismiss(); } catch(hash_exception) { tester.dismiss(); throw; }
I also have a similar class which tests the class invariants - in order to test basic exception safety.
This statement is self-conflicting or what the word.
It doesn't have a 'dismiss' method as I want to run the test regardless. This is why I would like to be able to use BOOST_CHECK in the destructor, but that's more of a convenience than anything.
I gather you do invariants checks in destructor of your scope guard like tester class. I don't think this is good idea to do anything smart in destructors (with rare exceptions). I don't really see why is it better then: tester t(x) try { // test case body } catch(...) { t.check_invariants(); throw; } Gennadiy
Gennadiy Rozental wrote:
"Daniel James" <daniel@calamity.org.uk> wrote in message news:doa5fo$fd6$1@sea.gmane.org...
Gennadiy Rozental wrote:
To check basic ES you don't need to catch any exceptions. Framework will check that you do unwinding properly by making sure you are not leaking any resources.
That isn't exactly true. The class still needs to meet it's invariants.
No. This is not what a definition says.
I think we're using jargon a bit differently here. When I'm writing about a class invariant, I mean the requirements that it must always meet - such as having a correct size. This is a pretty common usage. http://en.wikipedia.org/wiki/Class_invariant
For example, if an exception is thrown while rehashing a hash table, it will probably be left with only some of its elements - so the test needs to check that the size has been changed to match the decreased number of elements. Of course, that's possible within your framework.
This is requirement for strong guarantee
This is definitely the basic guarantee. From http://www.stlport.org/doc/exception_safety.html: "Containers continue to fulfill all of their requirements, even after an exception occurs during a mutating function. For example, a map will never give an inaccurate report of its size, or fail to meet its performance requirements because the tree that implements it has become unbalanced."
Another version would be to have two separate tests, the first to test for basic safety:
unordered_set<test::mock_object, test::hash, test::equal, test::allocator> x; x.insert(1);
and one to test for strong safety, with a non-throwing hash:
boost::unordered_set<test::mock_object, test::hash_nothrow, test::equal, test::allocator> x; unordered_strong_test tester(x); try { x.insert(1); } catch(...) { tester.test(); throw; }
This is a correct way of structuring your tests. Don't mix basic and strong test withing the same test case. TDD actually recommends to have separate test case for every single assertion. And in this design you do not need to know where exception was thrown from. It's possible that your class needs to check different invariants depending on where the failure occur. Do NOT do this within same test case. Use two - with different mocks throwing exceptions in different locations and different invariants checks.
That seems very dogmatic. Personally, I find the other styles more expressive. And while you might not need to know where the exception is thrown from, you need to control where it can be thrown from, which seems like more of an effort to me. Some of the other exception specifications details which methods exceptions can be thrown from, so you need quite fine-grained control. Daniel
For example, if an exception is thrown while rehashing a hash table, it will probably be left with only some of its elements - so the test needs to check that the size has been changed to match the decreased number of elements. Of course, that's possible within your framework.
This is requirement for strong guarantee
This is definitely the basic guarantee. From http://www.stlport.org/doc/exception_safety.html:
"Containers continue to fulfill all of their requirements, even after an exception occurs during a mutating function. For example, a map will never give an inaccurate report of its size, or fail to meet its performance requirements because the tree that implements it has become unbalanced."
Ok. you right.
Another version would be to have two separate tests, the first to test for basic safety:
unordered_set<test::mock_object, test::hash, test::equal, test::allocator> x; x.insert(1);
and one to test for strong safety, with a non-throwing hash:
boost::unordered_set<test::mock_object, test::hash_nothrow, test::equal, test::allocator> x; unordered_strong_test tester(x); try { x.insert(1); } catch(...) { tester.test(); throw; }
This is a correct way of structuring your tests. Don't mix basic and strong test withing the same test case. TDD actually recommends to have separate test case for every single assertion. And in this design you do not need to know where exception was thrown from. It's possible that your class needs to check different invariants depending on where the failure occur. Do NOT do this within same test case. Use two - with different mocks throwing exceptions in different locations and different invariants checks.
That seems very dogmatic. Personally, I find the other styles more expressive. And while you might not need to know where the exception is thrown from, you need to control where it can be thrown from, which seems like more of an effort to me.
I guess it could be a matter of preference. I could try to support alternative that throw user supplied exception. There are several complications: 1. Macro does not support varargs. That mean We need add new macro into interface 2. Currently all methods are virtual. I couldn't pass custom user exception throw non-template virtual call.
Some of the other exception specifications details which methods exceptions can be thrown from, so you need quite fine-grained control.
Sorry I did not get this statement. Gennadiy P.S. Other then issues we discussing now, what is you general impression of new facilities?
Gennadiy Rozental wrote:
I guess it could be a matter of preference. I could try to support alternative that throw user supplied exception. There are several complications:
1. Macro does not support varargs. That mean We need add new macro into interface 2. Currently all methods are virtual. I couldn't pass custom user exception throw non-template virtual call.
Would it be possible to supply a templated exception that could be used. Something like: struct hash_exception_tag {}; typedef boost::itest::exception<hash_tag> hash_exception; // An exception derived from hash_exception struct hash_constructor_exception_tag {}; typedef boost::itest::exception<hash_constructor_tag, hash_exception> hash_constructor_exception; Would that be easier? Although, I get the impression that making use of your scope tracking might be easier.
Some of the other exception specifications details which methods exceptions can be thrown from, so you need quite fine-grained control.
Sorry I did not get this statement.
I was referring to something like: "For unordered associative containers, no swap function throws an exception unless that exception is thrown by the copy constructor or copy assignment operator of the container’s Hash or Pred object (if any)." Which is from TR1 6.3.1.1/3. To be honest, this one is pretty easy to deal with, but I could imagine more difficult ones.
P.S. Other then issues we discussing now, what is you general impression of new facilities?
Sorry, I always concentrate on the negative. It looks very powerful. It does a lot more than my test framework does and is better designed (my tests were thrown together as I went along). And it found a bug in one of my tests - I was using std::rand() but forgot to reset the seed at the beginning of each iteration, and your framework noticed that it was acting inconsistently. It will be interesting to see how this works in practice. I think most of my problems came from trying to migrate my tests from my framework to yours. And the specification for unordered associative containers is fairly demanding, although with the increasing popularity of policy templates that could become more common. I'll continue with it and let you know how I get on. I forgot to answer this before:
I realize that global operator new overload bound to cause some false positives under some circumstances. Users will need to address this in their code. If you have any better proposition I am all for it.
boost::itest::exception_safety could take an extra parameter which takes flags to turn of parts of the testing tracking (although that depends on how easy it is to seperate them). Maybe something like: boost::itest::exception_safety(&insert_test, "Insert Test", boost::itest::no_leak_detection); I don't think there's any need to make this accessible through the macro. Daniel
Although, I get the impression that making use of your scope tracking might be easier.
How about if I put itest::exception into public interface and make it aware about scope it was thrown from?
I realize that global operator new overload bound to cause some false positives under some circumstances. Users will need to address this in their code. If you have any better proposition I am all for it.
boost::itest::exception_safety could take an extra parameter which takes flags to turn of parts of the testing tracking (although that depends on how easy it is to seperate them). Maybe something like:
boost::itest::exception_safety(&insert_test, "Insert Test", boost::itest::no_leak_detection);
I don't think there's any need to make this accessible through the macro.
I could turn off leaks detection at runtime. But: 1. You could do this at compile time for both library and test or only test if you are using included components 2. It's going to be user's responsibility to check for leaks *and* notify the framework (using BOOST_ERROR for example) so that it could report failed execution path. Does it worth it? Gennadiy
Gennadiy Rozental wrote:
Although, I get the impression that making use of your scope tracking might be easier.
How about if I put itest::exception into public interface and make it aware about scope it was thrown from?
Yes, I think that would work.
I could turn off leaks detection at runtime. But:
1. You could do this at compile time for both library and test or only test if you are using included components
I'm not keen on this - especially for tests that are run as part of boost regression testing as it's nice to use the shared libraries. I'm not as keen on header only libraries as many on this list.
2. It's going to be user's responsibility to check for leaks *and* notify the framework (using BOOST_ERROR for example) so that it could report failed execution path.
Does it worth it?
Well, I think so, but I'm not 100% sure. I suppose my concern is that most of Boost.Test is very flexible and extensible, while the exception testing seems quite closed in comparison. But, of course, it's at an early stage and it might be best to get some more experience with it and see how it goes. Daniel
Daniel James <daniel@calamity.org.uk> writes:
Gennadiy Rozental wrote:
It would be good to able to use my own exception types instead of Boost.Test's private exception type. You could possibly write a version of BOOST_ITEST_EPOINT which lets the user specify their own exception type, and some kind of mechanism to specify which exceptions are allowed seems essential.
Why would you need an ability to specify which fake exception to throw? This facility tests namely this ability of your code to do unwinding properly in case of any exception in any place. You don't really need to know which exception it was. To check post exception invariants you could use catch(...)
Unordered associative containers have a few exception specifications like (from TR1 6.3.1.1/2):
"For unordered associative containers, if an exception is thrown by any operation other than the container’s hash function from within an insert() function inserting a single element, the insert() function has no effect."
Careful with your terminology. In the standard, "exception specification" means something else. Or at least, "exception-specification" does. -- Dave Abrahams Boost Consulting www.boost-consulting.com
participants (3)
-
Daniel James -
David Abrahams -
Gennadiy Rozental