Testing and toolsets
Hi Everyone, Could anyone give me some information on how regression test matrix is supposed to work? I can see that when testers run their tests they specify a toolset; this toolset is reported in test matrix, and based on this toolset name failures in tests can be labeled as expected. Did I get it right? However, the fact that a tester turns C++11 support or not, is not necessarily reflected in the toolset name (some testers indicate it in the toolset name, others don't). On the other hand C++11 being turned on or off can affect the test results. In that case the toolset name alone is not sufficient to make a decision whether I expect a failure or not. Is there some policy for addressing this problem? Am I the only person to be affected? Regards, &rzej
On Monday 19 January 2015 15:14:21 Andrzej Krzemienski wrote:
Hi Everyone, Could anyone give me some information on how regression test matrix is supposed to work? I can see that when testers run their tests they specify a toolset; this toolset is reported in test matrix, and based on this toolset name failures in tests can be labeled as expected. Did I get it right?
However, the fact that a tester turns C++11 support or not, is not necessarily reflected in the toolset name (some testers indicate it in the toolset name, others don't). On the other hand C++11 being turned on or off can affect the test results. In that case the toolset name alone is not sufficient to make a decision whether I expect a failure or not.
Is there some policy for addressing this problem? Am I the only person to be affected?
AFAIK, there is no formal property of a toolset that would differentiate testers between C++ versions. You basically rely on toolset names. Unfortunately, some testers reflect C++ version in the tester name rather than the toolset, and I don't think there's a way to use the tester name in the markup table. I would kindly ask those testers to change their settings. If you just want to see the tester description you can read it through the link which is the tester name in the matrix. Some testers provide more detailed info there.
On Monday, January 19, 2015 10:47 PM, Andrey Semashev wrote:
On Monday 19 January 2015 15:14:21 Andrzej Krzemienski wrote:
Hi Everyone, Could anyone give me some information on how regression test matrix is supposed to work? I can see that when testers run their tests they specify a toolset; this toolset is reported in test matrix, and based on this toolset name failures in tests can be labeled as expected. Did I get it right?
However, the fact that a tester turns C++11 support or not, is not necessarily reflected in the toolset name (some testers indicate it in the toolset name, others don't). On the other hand C++11 being turned on or off can affect the test results. In that case the toolset name alone is not sufficient to make a decision whether I expect a failure or not.
Is there some policy for addressing this problem? Am I the only person to be affected?
AFAIK, there is no formal property of a toolset that would differentiate testers between C++ versions. You basically rely on toolset names. Unfortunately, some testers reflect C++ version in the tester name rather than the toolset, and I don't think there's a way to use the tester name in the markup table. I would kindly ask those testers to change their settings.
If you just want to see the tester description you can read it through the link which is the tester name in the matrix. Some testers provide more detailed info there.
Should there be a C++ version in the list of build features? Would that help? http://www.boost.org/build/doc/html/bbv2/overview/builtins/features.html I'm more than happy to change anything with my configuration, although I think enough details are there, if any more are needed, or the formatting can be improved, feel free to make suggestions. Ben (BP x86_64 C++11) - the name is no longer correct, but changing it means somebody has to delete the old results file)
On Monday 19 January 2015 23:03:37 Ben Pope wrote:
On Monday, January 19, 2015 10:47 PM, Andrey Semashev wrote:
AFAIK, there is no formal property of a toolset that would differentiate testers between C++ versions. You basically rely on toolset names. Unfortunately, some testers reflect C++ version in the tester name rather than the toolset, and I don't think there's a way to use the tester name in the markup table. I would kindly ask those testers to change their settings.
If you just want to see the tester description you can read it through the link which is the tester name in the matrix. Some testers provide more detailed info there.
Should there be a C++ version in the list of build features? Would that help?
http://www.boost.org/build/doc/html/bbv2/overview/builtins/features.html
I'm more than happy to change anything with my configuration, although I think enough details are there, if any more are needed, or the formatting can be improved, feel free to make suggestions.
Boost.Build feature for C++ version would be nice, but I don't think this would help the markup. If you look at the markup file[1], you'll notice that one can only identify toolsets by their names, not even by tester names. So unless format of that file changes, it's preferable if the toolset name reflects its configuration. [1] https://github.com/boostorg/boost/blob/master/status/explicit-failures-marku...
Andrey Semashev-2 wrote
AFAIK, there is no formal property of a toolset that would differentiate testers between C++ versions. You basically rely on toolset names.
When I run tests on my local system with gcc or clang I use the compile time switch -std=c++03 or -std=c++11 on my bjam command line which gets passed on to the gcc/clang command line. I started doing this when it occurred to me that I didn't no which version of C++ my library was being tested with. I still don't know what happens if you include no such switch at all. Many testers are including this information on the test properties page others are either not including the information or are not specifying in which case it's a mystery what version of the compiler is being used. In my test programs - I use boost config macros to detect and run/not run those parts of the test programs which are relevant to this or that compiler version. I would like a method to condition an entire test program on the value of a boost config macro. I realize that this is a pain and god forbid I have to make my bjam script even more complex than it is or have to remember some more boost build syntax/rules. So far I've been able to avoid needing this.
Unfortunately, some testers reflect C++ version in the tester name rather than the toolset, and I don't think there's a way to use the tester name in the markup table. I would kindly ask those testers to change their settings.
+1 a very easy, convenient and expedient fix. I'm sure the testers will accomodate a polite request to do this. I might add that I'd like to see all tests use an explicit -std=C++.. switch.
If you just want to see the tester description you can read it through the link which is the tester name in the matrix. Some testers provide more detailed info there.
I would also like to see more information there. -- View this message in context: http://boost.2283326.n4.nabble.com/Testing-and-toolsets-tp4671360p4671369.ht... Sent from the Boost - Dev mailing list archive at Nabble.com.
Robert Ramey wrote:
When I run tests on my local system with gcc or clang I use the compile time switch -std=c++03 or -std=c++11 on my bjam command line which gets passed on to the gcc/clang command line.
This isn't very convenient, because you can't run the 03 and 11 tests at the same time (and they go into the same directory). What I use is this (in user-config.jam): import feature ; import toolset ; using gcc ; feature.subfeature toolset gcc : std : cxx0x cxx11 cxx1y cxx14 : optional composite propagated ; feature.compose <toolset-gcc:std>cxx0x : <cxxflags>-std=c++0x ; feature.compose <toolset-gcc:std>cxx11 : <cxxflags>-std=c++11 ; feature.compose <toolset-gcc:std>cxx1y : <cxxflags>-std=c++1y ; feature.compose <toolset-gcc:std>cxx14 : <cxxflags>-std=c++14 ; using clang : : : <cxxflags>-Wno-variadic-macros <cxxflags>-Wno-c99-extensions ; feature.subfeature toolset clang : std : cxx0x cxx11 cxx1y cxx14 : optional composite propagated ; feature.compose <toolset-clang:std>cxx0x : <cxxflags>-std=c++0x ; feature.compose <toolset-clang:std>cxx11 : <cxxflags>-std=c++11 ; feature.compose <toolset-clang:std>cxx1y : <cxxflags>-std=c++1y ; feature.compose <toolset-clang:std>cxx14 : <cxxflags>-std=c++14 ; This allows me to run b2 toolset=gcc,gcc-cxx11,clang,clang-cxx11 for example. -std=c++03 is, I believe, the default, but you can add it to the above if you like.
Peter Dimov-2 wrote
<snip> This allows me to run
b2 toolset=gcc,gcc-cxx11,clang,clang-cxx11
for example.
-std=c++03 is, I believe, the default, but you can add it to the above if you like.
Very useful information - I think it summarizes what I want to do but it also shows what drives me nuts about boost build and boost tools in general. It's very, very, difficult and time consuming to find such information. Robert Ramey -- View this message in context: http://boost.2283326.n4.nabble.com/Testing-and-toolsets-tp4671360p4671381.ht... Sent from the Boost - Dev mailing list archive at Nabble.com.
On 1/19/2015 11:14 AM, Robert Ramey wrote:
Andrey Semashev-2 wrote
AFAIK, there is no formal property of a toolset that would differentiate testers between C++ versions. You basically rely on toolset names.
When I run tests on my local system with gcc or clang I use the compile time switch -std=c++03 or -std=c++11 on my bjam command line which gets passed on to the gcc/clang command line. I started doing this when it occurred to me that I didn't no which version of C++ my library was being tested with. I still don't know what happens if you include no such switch at all.
Many testers are including this information on the test properties page others are either not including the information or are not specifying in which case it's a mystery what version of the compiler is being used.
In my test programs - I use boost config macros to detect and run/not run those parts of the test programs which are relevant to this or that compiler version.
I would like a method to condition an entire test program on the value of a boost config macro. I realize that this is a pain and god forbid I have to make my bjam script even more complex than it is or have to remember some more boost build syntax/rules. So far I've been able to avoid needing this.
Did you not see a previous post from John Maddock explaining how to use a "requires" rule invocation so that you can run a test depending on config macro features ? Does that do what you want ?
Edward Diener-3 wrote
On 1/19/2015 11:14 AM, Robert Ramey wrote:
I would like a method to condition an entire test program on the value of a boost config macro. I realize that this is a pain and god forbid I have to make my bjam script even more complex than it is or have to remember some more boost build syntax/rules. So far I've been able to avoid needing this.
Did you not see a previous post from John Maddock explaining how to use a "requires" rule invocation so that you can run a test depending on config macro features ?
No
Does that do what you want ?
Maybe. What I really want is a window into the boost build documention which people can post this extra information to. Sort of like php - documentation or similar to the pages in the boost library incubator (www.blincubator). So that I don't have to spend hours trolling the whole world's internet to find information like this. Robert Ramey -- View this message in context: http://boost.2283326.n4.nabble.com/Testing-and-toolsets-tp4671360p4671382.ht... Sent from the Boost - Dev mailing list archive at Nabble.com.
On 01/19/2015 09:34 PM, Robert Ramey wrote:
Edward Diener-3 wrote
On 1/19/2015 11:14 AM, Robert Ramey wrote:
I would like a method to condition an entire test program on the value of a boost config macro. I realize that this is a pain and god forbid I have to make my bjam script even more complex than it is or have to remember some more boost build syntax/rules. So far I've been able to avoid needing this.
Did you not see a previous post from John Maddock explaining how to use a "requires" rule invocation so that you can run a test depending on config macro features ?
No
Does that do what you want ?
Maybe.
What I really want is a window into the boost build documention which people can post this extra information to. Sort of like php - documentation or similar to the pages in the boost library incubator (www.blincubator). So that I don't have to spend hours trolling the whole world's internet to find information like this.
Robert, the standard way for projects on github to maintain such dynamic documentation is the wiki, so for Boost.Build, that would be: https://github.com/boostorg/build/wiki that page is empty now, but if anybody has content to add, it's one click. For example of a project that uses wiki for all documentation, see: https://github.com/mbostock/d3/wiki/API-Reference -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com
On 1/19/2015 1:34 PM, Robert Ramey wrote:
Edward Diener-3 wrote
On 1/19/2015 11:14 AM, Robert Ramey wrote:
I would like a method to condition an entire test program on the value of a boost config macro. I realize that this is a pain and god forbid I have to make my bjam script even more complex than it is or have to remember some more boost build syntax/rules. So far I've been able to avoid needing this.
Did you not see a previous post from John Maddock explaining how to use a "requires" rule invocation so that you can run a test depending on config macro features ?
No
Its title is: "[Config] Announce: New Build time configuration feature"
2015-01-19 23:35 GMT+01:00 Edward Diener <eldiener@tropicsoft.com>:
On 1/19/2015 1:34 PM, Robert Ramey wrote:
Edward Diener-3 wrote
On 1/19/2015 11:14 AM, Robert Ramey wrote:
I would like a method to condition an entire test program on the value of a boost config macro. I realize that this is a pain and god forbid I have to make my bjam script even more complex than it is or have to remember some more boost build syntax/rules. So far I've been able to avoid needing this.
Did you not see a previous post from John Maddock explaining how to use a "requires" rule invocation so that you can run a test depending on config macro features ?
No
Its title is:
"[Config] Announce: New Build time configuration feature"
Ok, got it. I have two questions, though. 1. Is any of the Boost libraries using it currently for regression tests, so that I can observe it in the wild? 2. Is this technique also recommended for my particular case of Optional references? The reason my tests fail on some (many) compilers is because different compilers have different bugs in the implementation of reference binding. Last time I was advised not to add macros into Boost.Config that only cover up for compiler bugs and are applicable to a single library. So I am not sure anymore what is the advice for the case of regression testing of Boost.Optional that is hit by reference binding bugs? Regards, Andrzej
Consider this my view as testing manager.. On Mon, Jan 19, 2015 at 8:14 AM, Andrzej Krzemienski <akrzemi1@gmail.com> wrote:
Hi Everyone, Could anyone give me some information on how regression test matrix is supposed to work? I can see that when testers run their tests they specify a toolset; this toolset is reported in test matrix, and based on this toolset name failures in tests can be labeled as expected. Did I get it right?
Correct. However, the fact that a tester turns C++11 support or not, is not
necessarily reflected in the toolset name (some testers indicate it in the toolset name, others don't). On the other hand C++11 being turned on or off can affect the test results. In that case the toolset name alone is not sufficient to make a decision whether I expect a failure or not.
Right. Is there some policy for addressing this problem? Am I the only person to
be affected?
My view is that you should not use the toolset name as a basis for markup. In an ideal world the status markup would entirely go away. As it's an inferior mechanism for dealing with expected test variance. Instead please use the config method as previously (and below) mentioned. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
2015-01-19 18:29 GMT+01:00 Rene Rivera <grafikrobot@gmail.com>:
Consider this my view as testing manager..
On Mon, Jan 19, 2015 at 8:14 AM, Andrzej Krzemienski <akrzemi1@gmail.com> wrote:
Hi Everyone, Could anyone give me some information on how regression test matrix is supposed to work? I can see that when testers run their tests they specify a toolset; this toolset is reported in test matrix, and based on this toolset name failures in tests can be labeled as expected. Did I get it right?
Correct.
necessarily reflected in the toolset name (some testers indicate it in
However, the fact that a tester turns C++11 support or not, is not the
toolset name, others don't). On the other hand C++11 being turned on or off can affect the test results. In that case the toolset name alone is not sufficient to make a decision whether I expect a failure or not.
Right.
Is there some policy for addressing this problem? Am I the only person to
be affected?
My view is that you should not use the toolset name as a basis for markup. In an ideal world the status markup would entirely go away. As it's an inferior mechanism for dealing with expected test variance. Instead please use the config method as previously (and below) mentioned.
I am sorry, I may be thinking slowly, but what config method are you referring to? Regards, &rzej
participants (8)
-
Andrey Semashev -
Andrzej Krzemienski -
Ben Pope -
Edward Diener -
Peter Dimov -
Rene Rivera -
Robert Ramey -
Vladimir Prus