[Boost-users] [Test] Test suite timeout.

18 Sep 2009

      Hello!

I have a question about timeouts implementation in Boost::Test.
Please take a look at a simple example I wrote to demonstrate the problem:

krivenok@develop2 16:50:52 /tmp/boost_test $ cat main.cpp
#include <boost/test/unit_test.hpp>

using namespace boost::unit_test;
using boost::unit_test::framework::master_test_suite;

////////////////////////////////////////////////////////////////////////////////

void some_test_case()
{
  BOOST_TEST_CHECKPOINT("About to enter an infinite loop!");
  while(1);
}

//////////////////////////////////////////////////////////////////////////////

void another_test_case()
{
  BOOST_TEST_CHECKPOINT("About to enter an infinite loop 2!");
  while(1);
}

//////////////////////////////////////////////////////////////////////////////

boost::unit_test::test_suite* create_test_suite()
{
  boost::unit_test::test_suite *ts = new 
boost::unit_test::test_suite("Some test suite");
  if(!ts) return 0;
  ts->add(BOOST_TEST_CASE(&some_test_case), 0, 5);
  ts->add(BOOST_TEST_CASE(&another_test_case), 0, 10);
  return ts;
}

///////////////////////////////////////////////////////////////////////////////

test_suite* TestSuitesInit(int argc, char* argv[])
{
  master_test_suite().add(create_test_suite(), 0, 3);
  return 0;
}

///////////////////////////////////////////////////////////////////////////////

int main(int argc, char** argv)
{
  return unit_test_main(&TestSuitesInit, argc, argv);
}

///////////////////////////////////////////////////////////////////////////////
krivenok@develop2 16:50:53 /tmp/boost_test $

I compiled it as follows:

krivenok@develop2 16:56:12 /tmp/boost_test $ g++-4.4.1 -Wall 
-I/usr/local/dev/boost-1.40.0/include main.cpp 
/usr/local/dev/boost-1.40.0/lib/libboost_unit_test_framework-mt.a
krivenok@develop2 16:56:15 /tmp/boost_test $

And the run:

krivenok@develop2 16:58:17 /tmp/boost_test $ time ./a.out
Running 2 test cases...
unknown location(0): fatal error in "some_test_case": signal: SIGALRM 
(timeout while executing function)
main.cpp(10): last checkpoint: About to enter an infinite loop!
unknown location(0): fatal error in "another_test_case": signal: SIGALRM 
(timeout while executing function)
main.cpp(18): last checkpoint: About to enter an infinite loop 2!

*** 2 failures detected in test suite "Master Test Suite"

real    0m15.002s
user    0m14.920s
sys     0m0.030s
krivenok@develop2 16:58:36 /tmp/boost_test $

According to timings measurement I concluded that Boost::Test supports 
timeouts on per-test-case basis.
Test suite timeout (3 sec) is ignored.

Is it a bug?

P.S.

I also found that timeouts are implemented using alarm system call:

krivenok@develop2 16:58:36 /tmp/boost_test $ strace -fF ./a.out 2>&1 | 
grep alarm
alarm(0)                                = 0
alarm(5)                                = 0
alarm(0)                                = 0
alarm(0)                                = 0
alarm(10)                               = 0
alarm(0)                                = 0
krivenok@develop2 17:02:03 /tmp/boost_test $

As I know you can't start two or more timers using alarm since alarm uses
exactly one internal timer.
The following simple program demonstrates the problem:

krivenok@develop2 17:06:38 /tmp/boost_test/alarm $ cat main.c
#include <unistd.h>

int main()
{
  alarm(2);
  alarm(5);
  while(1);
  return 0;
}
krivenok@develop2 17:06:42 /tmp/boost_test/alarm $ gcc-4.4.1 main.c
krivenok@develop2 17:06:44 /tmp/boost_test/alarm $ time ./a.out
Alarm clock

real    0m5.001s
user    0m4.990s
sys     0m0.000s
krivenok@develop2 17:06:51 /tmp/boost_test/alarm $ strace -fF ./a.out 
2>&1 | grep alarm
alarm(2)                                = 0
alarm(5)                                = 2
krivenok@develop2 17:07:00 /tmp/boost_test/alarm $

This makes the implementation of mixed timeouts difficult and unreliable.
For example, consider the following test tree:
MasterTestSuite (timeout = T0)
  -- TestSuite1 (timeout = T1)
          -- TestCase11(timeout = T11)
          -- TestCase12(timeout = T12)
  -- TestSuite2 (timeout = T2)
          -- TestCase21(timeout = T21)
          -- TestCase212(timeout = T22)

Actual timeout of any test case depends on timeouts of all nodes in the 
path.

-- 
Sincerely yours, Dmitry V. Krivenok
Orange System Co., Ltd.
Saint-Petersburg, Russia
work phone: +7 812 332-32-40
cellular phone: +7 921 576-70-91
e-mail: krivenok@orangesystem.ru
web: http://www.orangesystem.ru
skype: krivenok_dmitry
jabber: krivenok_dmitry@jabber.ru
icq: 242-526-443