Re: [boost] Re: [Boost.Test] New testing procedure?
Jeff Garland wrote:
Well as soon as Robert wants to run the torture test he's going to get it at all sites if he controls it via his Jamfile. So we need some boost-wide option to define these variations. Hopefully my other email clarifies the idea.
My concern was raised base on the following scenario: a) I get things working to taste on the 3 or 4 compilers I have installed. b) I check in to CVS c) Test is run - takes a long time d) But shows some problems in the 5 compilers I don't have e) Now I start to fix stuff - more or less one compiler at a time f) Now much computer resource are used to no good purpose. I can't stop the test from running on the compilers that I know will fail anyway. g) Now my interest shifts to another compiler and a whole different set of wasted effort is done. .... q) Now everything is spiffy r) test runs - but only those tests that are dependant on something that has changed. The situation is tolerable - until something changes that changes the library - then caos from q onward the problem might be manageable if BJAM was could manage dependencies from are particular x.hpp -> y.cpp -> test.cpp. But as it is now, if one *.cpp file in the library has to be rebuilt, all the test that depend on any portion of the library have to be run. I realize this is an unrealistic hope but we're allowed to dream. So checking in a new Jamfile to customize at test is OK but it still leaves me with doing all the compilers even though I know it's a waste of time. Robert Ramey
"Robert Ramey" <ramey@rrsd.com> writes:
Jeff Garland wrote:
Well as soon as Robert wants to run the torture test he's going to get it at all sites if he controls it via his Jamfile. So we need some boost-wide option to define these variations. Hopefully my other email clarifies the idea.
My concern was raised base on the following scenario:
a) I get things working to taste on the 3 or 4 compilers I have installed. b) I check in to CVS c) Test is run - takes a long time d) But shows some problems in the 5 compilers I don't have e) Now I start to fix stuff - more or less one compiler at a time f) Now much computer resource are used to no good purpose. I can't stop the test from running on the compilers that I know will fail anyway. g) Now my interest shifts to another compiler and a whole different set of wasted effort is done. ....
OK, but none of the features suggested so far are going to fix that. You just want to say "this library/test doesn't work on these compilers", and as you start to implement workarounds for each compiler, knock it off the list so it'll be tested. We currently already have a way of expressing that "known to not work on X compiler" idea in the regression system. We could make the build system smart about it. Probably simpler, though, you could just put something in one of your library headers that said: #if BOOST_WORKAROUND(compiler1) || BOOST_WORKAROUND(compiler2) || ... # error doesn't work yet #endif And the tests would take very little time when they are known to fail. Hmm, that would cause retesting of known-working configurations. I guess a change to the build system would be superior.
q) Now everything is spiffy r) test runs - but only those tests that are dependant on something that has changed. The situation is tolerable - until something changes that changes the library - then caos
from q onward the problem might be manageable if BJAM was could manage dependencies from are particular x.hpp -> y.cpp -> test.cpp.
Of course it can and does.
But as it is now, if one *.cpp file in the library has to be rebuilt, all the test that depend on any portion of the library have to be run.
Of course they should.
I realize this is an unrealistic hope but we're allowed to dream.
I clearly don't know what you're talking about here. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
David Abrahams writes:
"Robert Ramey" <ramey@rrsd.com> writes:
My concern was raised base on the following scenario:
a) I get things working to taste on the 3 or 4 compilers I have installed. b) I check in to CVS c) Test is run - takes a long time d) But shows some problems in the 5 compilers I don't have e) Now I start to fix stuff - more or less one compiler at a time f) Now much computer resource are used to no good purpose. I can't stop the test from running on the compilers that I know will fail anyway. g) Now my interest shifts to another compiler and a whole different set of wasted effort is done. ....
OK, but none of the features suggested so far are going to fix that. You just want to say "this library/test doesn't work on these compilers", and as you start to implement workarounds for each compiler, knock it off the list so it'll be tested. We currently already have a way of expressing that "known to not work on X compiler" idea in the regression system. We could make the build system smart about it.
Probably simpler, though, you could just put something in one of your library headers that said:
#if BOOST_WORKAROUND(compiler1) || BOOST_WORKAROUND(compiler2) || ... # error doesn't work yet #endif
And the tests would take very little time when they are known to fail. Hmm, that would cause retesting of known-working configurations. I guess a change to the build system would be superior.
It would be. What needs to be done to implement it? -- Aleksey Gurtovoy MetaCommunications Engineering
participants (3)
-
Aleksey Gurtovoy -
David Abrahams -
Robert Ramey