[Boost.Test] A possible feature: run only tests from recently modified sources
For my work I usually write an unit test and want to execute this test immediatelly to see whether and how it passes. If an application has many, many tests running them all would slow me down and disrupt the flow. To deal with this I once wrote small test library which checks timestamps of source files containing tests and runs only tests from recently modified files. (In my case I run tests from files changed during last 10 mins when the application starts.) This way I am able to write a test and run it within mere second(s). Perhaps similar functionality could be considered into Boost.Test (I didn't find it here). The implementation is portable, it uses only __FILE__, stat() and time(). It may be possible to check also source file dependencies for changes (up to some point) but I do not do it and it is not obstacle for me. Syntax I use looks like TEST { ... assert(...); // assert breaks into debugger } .. run_recently_modified_tests() /Pavel
Pavel Vozenilek wrote:
For my work I usually write an unit test and want to execute this test immediatelly to see whether and how it passes.
If an application has many, many tests running them all would slow me down and disrupt the flow.
To deal with this I once wrote small test library which checks timestamps of source files containing tests and runs only tests from recently modified files.
You already get that feature by integrating your tests with a build system. Possibly with full dependency checking as well. Is there something I'm missing? João
Syntax I use looks like
TEST { ... assert(...); // assert breaks into debugger }
.. run_recently_modified_tests()
You need to specify somehow what file each particular test case depend on, isn't it? Gennadiy
"Gennadiy Rozental" wrote:
Syntax I use looks like
TEST { ... assert(...); // assert breaks into debugger }
.. run_recently_modified_tests()
You need to specify somehow what file each particular test case depend on, isn't it?
The source file name is recorded as during registration of each test (during statics initialization). Then go through list of source files and execute only tests from those touched within last 10 minutes. /Pavel
You need to specify somehow what file each particular test case depend on, isn't it?
The source file name is recorded as during registration of each test (during statics initialization).
Then go through list of source files and execute only tests from those touched within last 10 minutes.
So essentially you looking for something like this: BOOST_AUTO_TEST_CASE( test_abc ) { .... } BOOST_TEST_CASE_CONDITION( test_abc, BOOST_TEST_CASE_FILE_DEPEND( abc.cpp, 10*60 ) ) ; We could design several different runtime conditions: file dependence, time dependency and provide API for custom users conditions. I will need to think how to implement something like this. Meanwhile if you could supply code for timestamp comparisons it'll be useful. Gennadiy
"Gennadiy Rozental" wrote:
So essentially you looking for something like this:
BOOST_AUTO_TEST_CASE( test_abc ) { .... }
BOOST_TEST_CASE_CONDITION( test_abc, BOOST_TEST_CASE_FILE_DEPEND( abc.cpp, 10*60 ) ) ;
We could design several different runtime conditions: file dependence, time dependency and provide API for custom users conditions.
I will need to think how to implement something like this. Meanwhile if you could supply code for timestamp comparisons it'll be useful.
http://www.codeproject.com/cpp/minitest.asp contains such code (one header file). It is just a call to stat(). /Pavel
I will need to think how to implement something like this. Meanwhile if you could supply code for timestamp comparisons it'll be useful.
http://www.codeproject.com/cpp/minitest.asp
contains such code (one header file). It is just a call to stat().
The problem I have with the whole idea is that it's more natural to achieve on makefile level and doesn't require too much efforts. Why would you want to bundle test for different units into single unit test module? Gennadiy
"Gennadiy Rozental" wrote
http://www.codeproject.com/cpp/minitest.asp
contains such code (one header file). It is just a call to stat().
The problem I have with the whole idea is that it's more natural to achieve on makefile level and doesn't require too much efforts.
I think we may be talking about two different use cases for test system: * one is to run them overnight, do a complete coverage, record everything, recover after errors * other is quick and dirty tool to test recently changed code directly from IDE, without need to touch makefiles or whatever else. Boost.Test may be able to serve both purposes.
Why would you want to bundle test for different units into single unit test module?
I do not understand this. /Pavel
Pavel Vozenilek a écrit :
I think we may be talking about two different use cases for test system:
* one is to run them overnight, do a complete coverage, record everything, recover after errors
* other is quick and dirty tool to test recently changed code directly from IDE, without need to touch makefiles or whatever else.
I'd like to introduce yet another scenario for boost::test: * once a bug has been found, it is possible to exercise the test case that exhibits the bug, and this test case only, until the bug is corrected. -- Loïc
"Loïc Joly" wrote: Pavel Vozenilek a écrit :
I think we may be talking about two different use cases for test system:
* one is to run them overnight, do a complete coverage, record everything, recover after errors
* other is quick and dirty tool to test recently changed code directly from IDE, without need to touch makefiles or whatever else.
I'd like to introduce yet another scenario for boost::test: * once a bug has been found, it is possible to exercise the test case that exhibits the bug, and this test case only, until the bug is corrected. ------------- There could be unwanted interdependencies between tests. I use random sorting of the tests before run with ability to specify the seed manually in case of such problem. (I happens only /very/ seldom I need it.) /Pavel
"Gennadiy Rozental" wrote:
You need to specify somehow what file each particular test case depend on, isn't it?
If you mean header dependencies: for my typical use this is not necessary. The feature helps me to quickly run test I wrote or get warm feeling that last changes didn't break something. The complete test run executes hundreths of tests and takes too much of time. I have been even toying with an idea to compile the tests as separate dynamic libraries (invisibly to the project) and be able to change or add them them during runtime so the application won't need to be restarted. /Pavel
participants (4)
-
Gennadiy Rozental -
João Abecasis -
Loïc Joly -
Pavel Vozenilek