[test] reporting problems from tests

31 Mar 2008

      Hi Gennadiy! 

I'm using Boost.Test 1.34.1. 

I have some questions: 

1) Is it possible to customize the output? 
As I see in boost/test/impl/compiler_log_formatter.ipp: 

void 
compiler_log_formatter::print_prefix( std::ostream& output, const_string file, 
std::size_t line ) 
{ 
    output << file << '(' << line << "): "; 
} 

This is MSVC compiler output syntax, and integrates with MSVS only. 
But many other compilers (at least gcc, suncc) and unix tools like grep print 
their messages in the form "filename:line: ", and there are many tools/IDEs 
supporting exactly this format but not "filename(line): ".

Is it possible to customize it? 
Maybe by using Boost.Format, or any kind of config #define, or anything else. 

2) timeout. 
You working with timeout using SIGALRM, but this interferes affects sleep(), 
alarm() etc. 
You can read about it here: 
http://www.opengroup.org/onlinepubs/000095399/functions/sleep.html 
To me, it looks dangerous as there are many applications calling sleep(). 
What do you think? 

3) --catch_system_errors 
Is it possible to set it to false by default? 
Everybody wants to have a core dump if the program crashed, the only scenario 
when cores are unwelcome - huge test suits with many-many binaries, when 
crashing tests can just exhaust disk space. But such huge test suits will be 
run from scripts where the number of parameters doesn't matter.

But when I run a single test manually - I (and everybody I know) expect it to 
crash, to have a possibility to dive into the core with the debugger.

You know the errors leading to crashes are usually very hard to reproduce, so 
it's quite frustrating when you lose a chance to understand an error just 
because you forgot to type this "--catch_system_errors=no" or forgot to setup 
environment variable.

I just faced this problem. The test crashed but no core was dumped. Immediate 
rerun gave no crash. So I just ran the test in cycle and only after two(!) 
hours it finally crashed.

That's why I'm voting for setting crash the default option.

For now I just went to the boost sources and changed the code to make "no" 
default option. 
But I'm looking for more elegant solution. 
For example, why not to place all default values to config defines like 
BOOST_TEST_DEFAULT_SOMETHING? 

4) I finally found the problem in the crashing test - it crashed inside 
Boost.Test due to parallel calls to Boost.Test checks.
Any plans to do anything with this? 

5) Printing objects in failed tests
For example, what if I have BOOST_CHECK_EQUAL(a,b) and there is no operator<< 
defined in the classes?
Or they're defined, but do something that I don't want to see in this 
particular test output (because it dumps the object in binary form, for 
example)?
Is it possible to customize this somehow, e.g. by passing a printing functor as 
a third parameter, or any other way?

Bests, 
Maxim

Maxim Yanchenko

Gennadiy Rozental

tags

participants (2)