Say a library now covers 100 test cases spread out among five test programs. I'd like to see those five programs refactored into one program, still covering the 100 test cases. Or even adding more test cases. A single program would cut the overhead, including the human overhead.
Sorry I missed the beginning of the thread: how would having one program help?
There are at least three drawbacks to this approach:
1. "something is wrong" is all the information you get from a failing test. Esp. you'll likely see only one of several problems related to a failing test program. The next problem will only become visible after the first problem has been fixed.
This is not necessarily true. If one uses Boost Unit test framework you most probably get all the errors in one shot.
2. Some tests are known to fail for certain compilers. If those tests are joined with other tests then we'll lose information about these other tests.
If it runtime errors you could rely on "expected errors" feature. You are right for compile time errors, though one could try to ifdef part of test program. Gennadiy.
Rozental, Gennadiy wrote:
There are at least three drawbacks to this approach:
1. "something is wrong" is all the information you get from a failing test. Esp. you'll likely see only one of several problems related to a failing test program. The next problem will only become visible after the first problem has been fixed.
This is not necessarily true. If one uses Boost Unit test framework you most probably get all the errors in one shot.
To clarify: I'm talking about compile time errors. Those worry me at lot more than runtime errors .
2. Some tests are known to fail for certain compilers. If those tests are joined with other tests then we'll lose information about these other tests.
If it runtime errors you could rely on "expected errors" feature. You are right for compile time errors, though one could try to ifdef part of test program.
I think #ifdeffing around known failures would already help with the existing tests. However, I'd prefer those compiler version checks to be in the build system. Regards, m
Martin Wille <mw8329@yahoo.com.au> writes:
2. Some tests are known to fail for certain compilers. If those tests are joined with other tests then we'll lose information about these other tests.
If it runtime errors you could rely on "expected errors" feature. You are right for compile time errors, though one could try to ifdef part of test program.
I think #ifdeffing around known failures would already help with the existing tests.
The problem is that the tests would "appear" to work to people looking at the regression logs, and if in fact they ever did start working, we'll never know that the tests can be re-enabled.
However, I'd prefer those compiler version checks to be in the build system.
Could you be more specific? -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Martin Wille <mw8329@yahoo.com.au> writes:
2. Some tests are known to fail for certain compilers. If those tests are joined with other tests then we'll lose information about these other tests.
If it runtime errors you could rely on "expected errors" feature. You are right for compile time errors, though one could try to ifdef part of test program.
I think #ifdeffing around known failures would already help with the existing tests.
The problem is that the tests would "appear" to work to people looking at the regression logs, and if in fact they ever did start working, we'll never know that the tests can be re-enabled.
However, I'd prefer those compiler version checks to be in the build system.
Could you be more specific?
I think you misunderstood me. The tests should not simply be skipped. Instead the should not be compiled/run but marked as failure. The Jamfile for a test could contain a check which OS/compiler the test should be run for and report a failure immediately for OSs/compilers the test is known to fail. Doing so would be ugly, of course, and it would be a tedious job but it would make the test procedure faster and expected test failures would also be documented automatically. We could also make those checks depend on an additional flag. Setting the flag would disallow the skipping of tests. With this feature we should be able to check for tests that (unexpectedly) start to work again. Regards, m
participants (3)
-
David Abrahams -
Martin Wille -
Rozental, Gennadiy