[config] [regression] Regression test failures due to compiler bugs -- your advice needed

Hi Everyone,
I need the advice from the experts, on what is the recommended way to
handle the following situation with regression tests.
A number of tests of Boost.Optional fail. (see here:
http://www.boost.org/development/tests/develop/developer/optional.html)
These are the failures in optional

Andrzej Krzemienski wrote:
A number of tests of Boost.Optional fail. (see here: http://www.boost.org/development/tests/develop/developer/optional.html) These are the failures in optional
(optional references), because quite a lot of compilers implement reference binding in a non-standard-conforming manner. (For details of these bugs see here: http://www.boost.org/doc/libs/develop/libs/optional/doc/html/boost_optional/... ).
On the doc page, you state that these bugs are present in all MSVC versions, but on the tests page, I only see one failure, with msvc-10.0, everything else is green (or, in some cases, missing.)

2015-01-21 16:41 GMT+01:00 Peter Dimov
Andrzej Krzemienski wrote:
A number of tests of Boost.Optional fail. (see here: http://www.boost.org/development/tests/develop/developer/optional.html) These are the failures in optional
(optional references), because quite a lot of compilers implement reference binding in a non-standard-conforming manner. (For details of these bugs see here: http://www.boost.org/doc/libs/develop/libs/optional/doc/ html/boost_optional/dependencies_and_portability/ optional_reference_binding.html ). On the doc page, you state that these bugs are present in all MSVC versions, but on the tests page, I only see one failure, with msvc-10.0, everything else is green (or, in some cases, missing.)
You are right. I need to check if I didn't loose some tests. But back to my original request, did what I say makes sense?

Andrzej Krzemienski wrote:
2015-01-21 16:41 GMT+01:00 Peter Dimov
: On the doc page, you state that these bugs are present in all MSVC versions, but on the tests page, I only see one failure, with msvc-10.0, everything else is green (or, in some cases, missing.)
You are right. I need to check if I didn't loose some tests. But back to my original request, did what I say makes sense?
I'd like to know more about what is failing where before I venture an
opinion about your original request.
An optional

2015-01-21 21:56 GMT+01:00 Peter Dimov
Andrzej Krzemienski wrote:
2015-01-21 16:41 GMT+01:00 Peter Dimov
: On the doc page, you state that these bugs are present in all MSVC > versions, but on the tests page, I only see one failure, with msvc-10.0, > everything else is green (or, in some cases, missing.)
You are right. I need to check if I didn't loose some tests. But back to my original request, did what I say makes sense?
I'd like to know more about what is failing where before I venture an opinion about your original request.
An optional
that fails silently for certain T's in a quite insidious way on a wide array of popular compilers, including all versions of MSVC, should perhaps be disabled for those T's outright, instead of the failures being marked. Users do not consult the markup when using optional<> and even if they did, optional is most likely to occur in generic code, not on its own. This will result in some very hard to track bugs.
Are you saying that on a compiler where an operation on optional

On Thu, Jan 22, 2015 at 12:10 AM, Andrzej Krzemienski
2015-01-21 21:56 GMT+01:00 Peter Dimov
: An optional
that fails silently for certain T's in a quite insidious way on a wide array of popular compilers, including all versions of MSVC, should perhaps be disabled for those T's outright, instead of the failures being marked. Users do not consult the markup when using optional<> and even if they did, optional is most likely to occur in generic code, not on its own. This will result in some very hard to track bugs. Are you saying that on a compiler where an operation on optional
might give wrong run-time result, the operation should be disabled (render a compile-time error)? If so, I agree, and intend to do it at some point. But even then I should indicate somehow in regression tests that I expect different results in gcc and in msvc.
What kind of errors are we talking about? Does optional

2015-01-21 22:14 GMT+01:00 Andrey Semashev
On Thu, Jan 22, 2015 at 12:10 AM, Andrzej Krzemienski
wrote: 2015-01-21 21:56 GMT+01:00 Peter Dimov
: An optional
that fails silently for certain T's in a quite insidious way on a wide array of popular compilers, including all versions of MSVC,
should perhaps be disabled for those T's outright, instead of the failures being marked. Users do not consult the markup when using optional<> and even if they did, optional
is most likely to occur in generic code, not on its own. This will result in some very hard to track bugs. Are you saying that on a compiler where an operation on optional
might give wrong run-time result, the operation should be disabled (render a compile-time error)? If so, I agree, and intend to do it at some point. But even then I should indicate somehow in regression tests that I expect different results in gcc and in msvc. What kind of errors are we talking about? Does optional
behave plain wrong on MSVC or is there a misbehavior in some narrow case? Does it compile invalid code, does it not compile valid one, or is there a miscompilation?
In most of the cases the program compiles but we have a run-time misbehavior: we store a reference to an unexpected temporary rather than to the original object.

On Thu, Jan 22, 2015 at 12:20 AM, Andrzej Krzemienski
2015-01-21 22:14 GMT+01:00 Andrey Semashev
: What kind of errors are we talking about? Does optional
behave plain wrong on MSVC or is there a misbehavior in some narrow case? Does it compile invalid code, does it not compile valid one, or is there a miscompilation? In most of the cases the program compiles but we have a run-time misbehavior: we store a reference to an unexpected temporary rather than to the original object.
Does it happen all the time or in some select cases? E.g. if binding a
reference to an integer literal doesn't work but everything else does
it's not a big deal since this rarely happens.
I'm kind of worried by disabling the feature outright. If I'm writing
portable code which uses optional

Andrzej Krzemienski wrote:
Are you saying that on a compiler where an operation on optional
might give wrong run-time result, the operation should be disabled (render a compile-time error)?
I think that am I saying that if an operation on optional

On Thu, Jan 22, 2015 at 12:27 AM, Peter Dimov
Andrzej Krzemienski wrote:
Are you saying that on a compiler where an operation on optional
might give wrong run-time result, the operation should be disabled (render a compile-time error)? I think that am I saying that if an operation on optional
is apt to result in dangling pointers on many popular compilers, the operation should be disabled for all compilers for consistency, not just those that will exhibit the failure. This interface is obviously too dangerous to be of any use and, while it will work on some compilers, its use will not be portable.
It's also possible to make the feature optional, so that it can be enabled by a macro.

Andrey Semashev-2 wrote
On Thu, Jan 22, 2015 at 12:27 AM, Peter Dimov <
lists@
> wrote:
Andrzej Krzemienski wrote:
Are you saying that on a compiler where an operation on optional<T&> might give wrong run-time result, the operation should be disabled (render a compile-time error)?
I think that am I saying that if an operation on optional<T&> is apt to result in dangling pointers on many popular compilers, the operation should be disabled for all compilers for consistency, not just those that will exhibit the failure. This interface is obviously too dangerous to be of any use and, while it will work on some compilers, its use will not be portable.
It's also possible to make the feature optional, so that it can be enabled by a macro.
On a recent project I did come across this with optional references: we decided to do exactly that: disabled them with optional macro to re-enable them. Mind that optional references are easily replaceable by raw pointers -> yield almost the same syntax as optionals and are plain simple. -- View this message in context: http://boost.2283326.n4.nabble.com/config-regression-Regression-test-failure... Sent from the Boost - Dev mailing list archive at Nabble.com.

2015-01-21 20:43 GMT+01:00 Andrzej Krzemienski
2015-01-21 16:41 GMT+01:00 Peter Dimov
: Andrzej Krzemienski wrote:
A number of tests of Boost.Optional fail. (see here: http://www.boost.org/development/tests/develop/developer/optional.html) These are the failures in optional
(optional references), because quite a lot of compilers implement reference binding in a non-standard-conforming manner. (For details of these bugs see here: http://www.boost.org/doc/libs/develop/libs/optional/doc/ html/boost_optional/dependencies_and_portability/ optional_reference_binding.html ). On the doc page, you state that these bugs are present in all MSVC versions, but on the tests page, I only see one failure, with msvc-10.0, everything else is green (or, in some cases, missing.)
You are right. I need to check if I didn't loose some tests. But back to my original request, did what I say makes sense?
Indeed, while refactoring unit tests I skipped the tests for optional ref converting assignments. Now the tests look better. I mean, worse. - Andrzej

Le 21/01/15 10:11, Andrzej Krzemienski a écrit :
Hi Everyone, I need the advice from the experts, on what is the recommended way to handle the following situation with regression tests.
A number of tests of Boost.Optional fail. (see here: http://www.boost.org/development/tests/develop/developer/optional.html) These are the failures in optional
(optional references), because quite a lot of compilers implement reference binding in a non-standard-conforming manner. (For details of these bugs see here: http://www.boost.org/doc/libs/develop/libs/optional/doc/html/boost_optional/... ).
Andrzej, are these regressions since last version(s) with compilers that worked, or are these related to new features that works on only some compilers? Best, Vicente

2015-01-21 17:31 GMT+01:00 Vicente J. Botet Escriba < vicente.botet@wanadoo.fr>:
Le 21/01/15 10:11, Andrzej Krzemienski a écrit :
Hi Everyone, I need the advice from the experts, on what is the recommended way to handle the following situation with regression tests.
A number of tests of Boost.Optional fail. (see here: http://www.boost.org/development/tests/develop/developer/optional.html) These are the failures in optional
(optional references), because quite a lot of compilers implement reference binding in a non-standard-conforming manner. (For details of these bugs see here: http://www.boost.org/doc/libs/develop/libs/optional/doc/ html/boost_optional/dependencies_and_portability/ optional_reference_binding.html ). Andrzej, are these regressions since last version(s) with compilers that worked, or are these related to new features that works on only some compilers?
Best, Vicente
These failures have been there for as long as optional references exist. It is that I no longer mark them as "expected".

On Wed, Jan 21, 2015 at 12:11 PM, Andrzej Krzemienski
Hi Everyone, I need the advice from the experts, on what is the recommended way to handle the following situation with regression tests.
A number of tests of Boost.Optional fail. These are the failures in optional
(optional references), because quite a lot of compilers implement reference binding in a non-standard-conforming manner. The question is, how these testing failures should be reflected in regression testing configuration, so that they do not appear as bugs in the implementation of Boost.Optional?
I don't think there is an official guideline about this. The traditional way of handling this is to mark the tests as expected failures with an appropriate comment. But since you can't do this, your only option is to leave them as is, I suppose. At least everyone will be able to see that a certain compiler does not handle certain cases covered by tests. Using Boost.Config based checks to run the tests, I think, is aiming a different purpose. I didn't use it in my tests but my understanding is that it allows to set preconditions for the test to build and run. If those preconditions are not met, the test is not built (i.e. excluded from the matrix) or can be built differently. It's more like a configure script, which can be used to adjust the build depending on what the compiler supports. I think, there is a limited set of checks provided by Boost.Config, which is probably similar to those features it provide macros for. Of course, you can write your own checks, similar to Boost.Config, which will test the compiler for the features you need. That'll require a certain amount of Boost.Build knowledge. Whichever approach you choose, it is a good idea to reflect in the docs what compilers don't support optional references, if it's not done yet.
participants (5)
-
Andrey Semashev
-
Andrzej Krzemienski
-
formiaczek
-
Peter Dimov
-
Vicente J. Botet Escriba