
Hi, I would like to take this moment to share with you finally the modifications Boost.Test went through during last couple month. Most of Boost.Test components undergo a major or significant change in both design and implementation. I've strived to keep an interface unchanged as much as possible. The primary goals for this update were: * Make a unit test framework better suited for different 'runners' * Make unit test framework simpler internally by eliminating test cases hierarchy * Make test tools more uniform * Make auto unit test framework as usable as possible Following are details of what updated/changed specifically in each area of Boost.Test: I. Execution Monitor The execution monitor design changes from inheritance to delegation. If previously one needed to inherit from the boost::execution monitor to do a monitored run of a function, now one need to employ an instance of the boost::execution_monitor: boost::execution_monitor em; // register translators for custom user exception em.register_exception_translator<ExceptionType>( my_translator ); em.execute( function_to_execute ); function_to_execute should be any zero arity function or function object. To implement the polymorphic function handling I employ my own class callback (see utilities section). I did not want to introduce a dependency on Boost.Function. But in a future I plan to switch to tr1::function. In addition following features implemented: * Debugger presence detected, in which case catching system errors (SEH) is disabled unless explicitly enabled. For now only works for msvc family * Automatic detection of memory leaks introduced. For now only works for msvc family Unit test framework turns this feature on by default. User could manage it by specifying runtime parameter (in a form of command line argument or environment variable). For example test --detect_memory_leak=<leak_allocation_number> set leak_allocation_number to integer number: 0 - to disable detection of memory leaks; 1 - to enable detection of memory leaks; N>1 - to enable detection of memory leaks and set a system breakpoint at the moment of Nth memory allocation. For more on how to find N and some issues with usage of this feature see documentation II Unit test framework (UTF) This component of Boost.Test undergo a most dramatic changes in both design and implementation. Here are main modifications: 1. UTF doesn't employ test cases hierarchy anymore. There is single class test_case that model one testing unit and class test_suite that model a collection of test units. Previously UTF employed compound test cases (for example parameterized test case) which went through it's components (simple subtests) during testing run time. Now compound test present itself as test case generator which produce a collection if test_case instances during test tree construction time. One consequence of this is that user doesn't need to keep track of compound test case parameters is it used be. test_case now employs a polymorphic callback to a actual test function. As a result of this one could supply any zero arity function as an argument of BOOST_TEST_CASE. SO no extra support needed to be able to create boost::function or boost::bind based tests cases. 2 .Introduced generic mechanism to traverse the test tree. Anyone willing to do so just need to implement test_tree_visitor interface and employ traverse_test_tree. This allows to implement any pre/post processing over test tree. 3. Introduced a notion of framework. The framework is responcible for: a) initializing and providing access to master test suite b) providing access to current test case c) running testing from given test tree node (simple or compound). By default it runs from root - master test suite. Supports both sequential and random order. d) One may register any test observer agent (implements test_observer interface) and framework notifies it of all test events e) providing access to any test case based to unique test unit id Using the framework one could implement different test runners. UTF library supplies one console runner (unit_test_main.cpp). But it's possible to do one winmain based or anything else. For example GUI program that build test tree and then run tests by request from any point in test tree. 4. Unit test log interface changed to single line (used to be begin/end macros). Unit test log formatter interface changed. How it much simpler and straightforward. Two supplied implementation are separated. 5. Notion of unit test result that was responcible for result collection and reporting eliminated. Instead introduced independent notions: results_collector and results_reporter. Results reporting formatter interface made public. Also results_reporter now allows to change an output stream. 6. Progress monitor functionality (which used to be activated in log level log_progress) now is an independent test observer and implemented by class progress_monitor. 7. Test case template interface changed. Now user doesn't need to explicitly register meta class. In addition couple more features implemented: * Added support for test case timing * UTF made __cdecl - safe * Result report include skipped tests * io saver facilities employed to guard against undesirable modification in output streams formatting * template test case automatically generate name that includes type name * --detect_memory_leak and --random cla added III Test Tools The two major changes in Test Tools are: * Test Tools interfaces to provide 3 version for each tool: CHECK- reports an error if assertion failed WARN - reports a warning if assertion failed REQUIRE - reports an error if assertion failed and aborts test case execution * Test Tools implementation completely reworked. All tools implemented now through single vararg function In addition following modifications made: * CHECK_COLLECTION interface changed: it now expect 4 arguments * BITWISE_EQUAL renamed to CHECK_BITWISE_EQUAL; old name still provided, but is deprecated * CHECK_COLLECTION interface changed to use PP_SEQ and as a result support arbitrary(actually maximum is fixed but configurable) number of predicate arguments * most of templates eliminated speeding up test modules compilation * floating-point precision set to include all significant digits (actually the formula may require some rework since it obviously doesn't do a good job sometimes) * BOOST_CHECK/WARN/REQUIRE_SMALL test tool added * old test tools names are *deprecated* IV Auto unit test framework Interesting discussion with Noel Llopis and several user requests lead to several additions to the auto_unit_test.hpp. Here is a short descriptions: * BOOST_AUTO_UNIT_TEST renamed to BOOST_AUTO_TEST_CASE. Old name still provided but deprecated * BOOST_AUTO_TEST_CASE_TEMPLATE provides an ability to write a self-registering test case template * BOOST_FIXTURE_TEST_CASE provides an ability to write a self-registering test case with fixture * BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES provides an ability to specify a number of expected failures in any self-registering test case * BOOST_AUTO_TEST_SUITE and BOOST_AUTO_TEST_SUITE_END provide an ability to structure test cases into self-registering test suites V Included components Implementation moved from source files (.cpp) under lib/test/src into implementation files (.ipp) under boost/test/impl to eliminate dependency of included/minimal components on lib/test/src directory. VI Utilities Basic utilities that may have value outside of Boost.Test now separated into utils subdirectory. If there is an interest to promote any of them onto public boost level I am ready to discuss it. Here is a short descriptions (some of them existed before, but most are new): 1) algorithm.hpp Implements several variations of STL algorithms including 4-arg mismatch 2) assign_op.hpp Hook for overloadable assignment 3) callback.hpp Implements a polymorphic function callback facility (similar to boost::function). 4) class_properties.hpp Simple facility that mimic notion of read-only read-write properties in C++ classes 5) custom_manip.hpp Simple helpers for creating custom output manipulators 6) fixed_mapping.hpp Fixed sized mapping (initialized in constructor ) with specified invalid value. 7) foreach.hpp This is an abridged version of BOOST_FOREACH. The only difference is portability. 8) named_params.hpp Facility for named function parameters support. Even though Dave A. did not see a value in my suggestion during Boost.Parameter review period. I couldn't justify for myself using that inconvenient interface (IMO) and absence of optional parameters. This facility implements an alternative solution with different set of tradeoffs and features. It used in several places in Boost.Test. 9) nullstream.hpp I should probably deprecate this one , but more-io was not accepted. 10) rtti.hpp Simple alternative to compiler RTTI. Provides an access to unique type id at runtime. I plan to extend this to implement type_descr similar to what is described in "C++ template metaprogramming". 11) trivial_singleton.hpp As the name suggest this simple singleton helper. 12) wrap_stringstream.hpp Wraps strstream and stringstream (depends with one is present) to provide the unified interface 13) basic_cstring/ class template basic_cstring wraps raw memory pointers and provide std_string like interface 14) iterator/input_iterator_facade.hpp Helps to build input iterators 15) iterator/token_iterator.hpp Yet another token iterator 16) iterator/istream_line_iterator.hpp Iterate over istream and produce lines 17) runtime/ This is long promised framework for parsing CLA handing environment variable and parsing config files. This is an alternative solution to what boost::program_options present. Unfortunately I did not have enough time to incorporate it completely into Boost.Test main components, so the detailed description will follow once this will happened. Sorry, for babbling for so long. Any comments are greatly appreciated. Regard, Gennadiy

Hi Gennadiy, ----------------- From: "Gennadiy Rozental" <gennadiy.rozental@thomson.com>
I would like to take this moment to share with you finally the modifications Boost.Test went through during last couple month. Most of Boost.Test components undergo a major or significant change in both design and implementation. I've strived to keep an interface unchanged as much as possible. The primary goals for this update were: * Make a unit test framework better suited for different 'runners' * Make unit test framework simpler internally by eliminating test cases hierarchy * Make test tools more uniform * Make auto unit test framework as usable as possible
I'm delighted to see boost test is evolving. What is the state of revisions to the documentation and test examples as personally I am finding it quite hard going to look around the code and discover the changes from 1.32 that will make unit testing somewhat simpler. I found Noel Lopis' unit test framework discussion document and subsequent review/discussion about boost test here to be very useful and am keen to use the results of your hard work! I've looked at the latest CVS and can't find any updated 1.33 document release notes (your post here is a good start however.) or other guides to new functionality. Given the imminent release of 1.33 do you have plans for updating the documentation prior to 1.33? [It may be I simply haven't found it yet!] Given the Test library's importance to Boost and software development in general, is this major evolution of the design one that warrants a re-review? In particular, while I understand the need to reduce dependencies on the rest of Boost, there seem to be several instances of the creation of largely equivalent functionality. Would it be better to leverage the work of others and note the dependencies? Your to-do list sounds interesting as well. The incorporation of performance monitoring/profiling sounds very interesting as does the various logging library discussions that are going on.

I'm delighted to see boost test is evolving.
What is the state of revisions to the documentation and test examples as personally I am finding it quite hard going to look around the code and discover the changes from 1.32 that will make unit testing somewhat simpler.
If you see a need for more examples feel free to submit one. I am ready to incorporate them.
I found Noel Lopis' unit test framework discussion document and subsequent review/discussion about boost test here to be very useful and am keen to use the results of your hard work!
I think I managed to satisfy all of the requirements we discussed.
I've looked at the latest CVS and can't find any updated 1.33 document release notes (your post here is a good start however.) or other guides to new functionality. Given the imminent release of 1.33 do you have plans for updating the documentation prior to 1.33? [It may be I simply haven't found it yet!]
Yeah. I know. I know. I am trying to catch this train, while conductor announces next station. Good news is that interfaces did not change much. Bad news is that I really out of free time. I'll do my best.
Given the Test library's importance to Boost and software development in general, is this major evolution of the design one that warrants a re-review?
I don't know about full review, but In general it may be a good idea. Unfortunately by the time it will be reviewed it's going to be 1.35 already.
In particular, while I understand the need to reduce dependencies on the rest of Boost, there seem to be several instances of the creation of largely equivalent functionality. Would it be better to leverage the work of others and note the dependencies?
In some cases (like callback) I expect to switch to use tr1 version. In other cases (like runtime params framework and named function params) I do not see it (promoting) happening. Unless there is a general interest for this. Gennadiy

On 05/08/2005 03:46 AM, Gennadiy Rozental wrote:
Hi,
I would like to take this moment to share with you finally the modifications Boost.Test went through during last couple month. Most of Boost.Test [snip] VI Utilities
Basic utilities that may have value outside of Boost.Test now separated into utils subdirectory. If there is an interest to promote any of them onto [snip] Sorry, for babbling for so long. Any comments are greatly appreciated.
I've had a need for tracking a class' instances for some time now. I've used: http://cvs.sourceforge.net/viewcvs.py/boost-sandbox/boost-sandbox/boost/utility/object_tracked.hpp?rev=1.1&view=auto to do this and it has worked pretty good. It has been used to write tests for memory management smart pointers. Some of the policy_ptr tests could also use something like this. Your test utilities library would be a better place for it than where it is now. Any chance of moving it there?

I've had a need for tracking a class' instances for some time now. I've used:
Could you give two lines rationale for what it is doing and how it's doing it? Gennadiy

On 05/11/2005 10:45 AM, Gennadiy Rozental wrote:
I've had a need for tracking a class' instances for some time now. I've used:
Could you give two lines rationale for what it is doing and how it's doing it? The CTOR simply records each instance in a class set and the DTOR removes it. The superclass assigns a unique ID to the instance.
------------------- It's really very simple, but I've seen the equivalent used in several memory managements test; hence, I thought it would a candidate for inclusion.
participants (3)
-
Gennadiy Rozental
-
Larry Evans
-
Paul Baxter