Re: [boost] Testing and toolsets

2015-01-19 18:29 GMT+01:00 Rene Rivera :

Consider this my view as testing manager..

On Mon, Jan 19, 2015 at 8:14 AM, Andrzej Krzemienski wrote:

...

Hi Everyone, Could anyone give me some information on how regression test matrix is supposed to work? I can see that when testers run their tests they specify a toolset; this toolset is reported in test matrix, and based on this toolset name failures in tests can be labeled as expected. Did I get it right?

Correct.

...

necessarily reflected in the toolset name (some testers indicate it in

However, the fact that a tester turns C++11 support or not, is not the

...

toolset name, others don't). On the other hand C++11 being turned on or off can affect the test results. In that case the toolset name alone is not sufficient to make a decision whether I expect a failure or not.

Right.

Is there some policy for addressing this problem? Am I the only person to

...

be affected?

My view is that you should not use the toolset name as a basis for markup. In an ideal world the status markup would entirely go away. As it's an inferior mechanism for dealing with expected test variance. Instead please use the config method as previously (and below) mentioned.

I am sorry, I may be thinking slowly, but what config method are you referring to? Regards, &rzej