Hello!
I have a question about timeouts implementation in Boost::Test.
Please take a look at a simple example I wrote to demonstrate the problem:
krivenok@develop2 16:50:52 /tmp/boost_test $ cat main.cpp
#include
using namespace boost::unit_test;
using boost::unit_test::framework::master_test_suite;
////////////////////////////////////////////////////////////////////////////////
void some_test_case()
{
BOOST_TEST_CHECKPOINT("About to enter an infinite loop!");
while(1);
}
//////////////////////////////////////////////////////////////////////////////
void another_test_case()
{
BOOST_TEST_CHECKPOINT("About to enter an infinite loop 2!");
while(1);
}
//////////////////////////////////////////////////////////////////////////////
boost::unit_test::test_suite* create_test_suite()
{
boost::unit_test::test_suite *ts = new
boost::unit_test::test_suite("Some test suite");
if(!ts) return 0;
ts->add(BOOST_TEST_CASE(&some_test_case), 0, 5);
ts->add(BOOST_TEST_CASE(&another_test_case), 0, 10);
return ts;
}
///////////////////////////////////////////////////////////////////////////////
test_suite* TestSuitesInit(int argc, char* argv[])
{
master_test_suite().add(create_test_suite(), 0, 3);
return 0;
}
///////////////////////////////////////////////////////////////////////////////
int main(int argc, char** argv)
{
return unit_test_main(&TestSuitesInit, argc, argv);
}
///////////////////////////////////////////////////////////////////////////////
krivenok@develop2 16:50:53 /tmp/boost_test $
I compiled it as follows:
krivenok@develop2 16:56:12 /tmp/boost_test $ g++-4.4.1 -Wall
-I/usr/local/dev/boost-1.40.0/include main.cpp
/usr/local/dev/boost-1.40.0/lib/libboost_unit_test_framework-mt.a
krivenok@develop2 16:56:15 /tmp/boost_test $
And the run:
krivenok@develop2 16:58:17 /tmp/boost_test $ time ./a.out
Running 2 test cases...
unknown location(0): fatal error in "some_test_case": signal: SIGALRM
(timeout while executing function)
main.cpp(10): last checkpoint: About to enter an infinite loop!
unknown location(0): fatal error in "another_test_case": signal: SIGALRM
(timeout while executing function)
main.cpp(18): last checkpoint: About to enter an infinite loop 2!
*** 2 failures detected in test suite "Master Test Suite"
real 0m15.002s
user 0m14.920s
sys 0m0.030s
krivenok@develop2 16:58:36 /tmp/boost_test $
According to timings measurement I concluded that Boost::Test supports
timeouts on per-test-case basis.
Test suite timeout (3 sec) is ignored.
Is it a bug?
P.S.
I also found that timeouts are implemented using alarm system call:
krivenok@develop2 16:58:36 /tmp/boost_test $ strace -fF ./a.out 2>&1 |
grep alarm
alarm(0) = 0
alarm(5) = 0
alarm(0) = 0
alarm(0) = 0
alarm(10) = 0
alarm(0) = 0
krivenok@develop2 17:02:03 /tmp/boost_test $
As I know you can't start two or more timers using alarm since alarm uses
exactly one internal timer.
The following simple program demonstrates the problem:
krivenok@develop2 17:06:38 /tmp/boost_test/alarm $ cat main.c
#include
int main()
{
alarm(2);
alarm(5);
while(1);
return 0;
}
krivenok@develop2 17:06:42 /tmp/boost_test/alarm $ gcc-4.4.1 main.c
krivenok@develop2 17:06:44 /tmp/boost_test/alarm $ time ./a.out
Alarm clock
real 0m5.001s
user 0m4.990s
sys 0m0.000s
krivenok@develop2 17:06:51 /tmp/boost_test/alarm $ strace -fF ./a.out
2>&1 | grep alarm
alarm(2) = 0
alarm(5) = 2
krivenok@develop2 17:07:00 /tmp/boost_test/alarm $
This makes the implementation of mixed timeouts difficult and unreliable.
For example, consider the following test tree:
MasterTestSuite (timeout = T0)
-- TestSuite1 (timeout = T1)
-- TestCase11(timeout = T11)
-- TestCase12(timeout = T12)
-- TestSuite2 (timeout = T2)
-- TestCase21(timeout = T21)
-- TestCase212(timeout = T22)
Actual timeout of any test case depends on timeouts of all nodes in the
path.
--
Sincerely yours, Dmitry V. Krivenok
Orange System Co., Ltd.
Saint-Petersburg, Russia
work phone: +7 812 332-32-40
cellular phone: +7 921 576-70-91
e-mail: krivenok@orangesystem.ru
web: http://www.orangesystem.ru
skype: krivenok_dmitry
jabber: krivenok_dmitry@jabber.ru
icq: 242-526-443
