Vladimir Prus wrote:

There is at least a couple of test failures, where the right course of action depends on what we want tests to mean.

First example is one of Boost.Test failures on darwin. I'm using this just as illustration.

The test in question checks that Boost.Test captures division by zero. However, as explained by Tobias, on Power processors division by zero is not automatically detected, and the compiler should emit extra code for checks, and does not emit such code by default. So, the test fails.

We can take two approaches:

1. If a test is green, it means tested functionality is available for users, when using freshly downloaded Boost, with default compiler options.

2. If a test is green, there's some combination of compiler options, and other setup, where this functionality is available.

Approach (2) might be more interesting to the library authors -- in particular case it's interesting to check all the code for detecting division by zero in Boost.Test, provided division itself is checked by compiler.

At the same time, (1) might be more interesting for user. If a user sees green cell, he's likely to expect the library to work with default options and no effort on his part. If that's not the case, it's better to leave the test failing and describe how user can get the desired functionality.

Any comments?

I don't know what the "right" answer is, but in cases where specific functionality is unavailable on a particular compiler/platform, one option is to use #ifdefs to force the test to pass (make it a "no op") and issue a "note" that shows up in the matrix. See for example libs/math/log1p_expm1_test.cpp Except I don't see the note showing up in the test results - also confirmed by looking at the results for type_traits/is_function_test - has this feature got lost/removed somewhere? John.