Interpretation of failing prg_exec_fail2 regression test

Currently the prg_exec_fail2 regression test fails on IBM VisualAge but this depends on how you interpret it. This regression test is supposed (by its creator) to fail at runtime because of a detection of a division by zero. If the executable does not detect the division by zero, the regression test is reported to fail. AFAICT this is not a failure of the executable on IBM, this is just a property of the executable. The idea of the regression tests is to verify if there are no regressions (by definition) and they are also used to verify if the compiler and created exe react as required by the standard. Well AFAIK nowhere is required that a division by zero is detected. It's just that some processors detect it, others don't. But I don't think regression tests should not be used to test 'properties' of the compiler or executable. Because I for instance interpret a failing regression test as being an error of the compiler or exe. There is a way though to detect division by zero with VisualAge by setting a specific pragma. However this incurs a runtime penalty (for the extra error checking code) so we don't want to do this in general. So I can assure the test to run as expected but I don't want to cripple performance just for the sake of passing this regression test. I can see 2 solutions: 1) Or regression tests are not used anymore to test *properties* of the platform/compiler/exe 2) Or I include the pragma only for this regression test just for the sake of passing the test. I definitly prefer option 1 because I consider option 2 as a cheat. toon

Sorry for late response.
This regression test is supposed (by its creator) to fail at runtime because of a detection of a division by zero. If the executable does not detect the division by zero, the regression test is reported to fail. AFAICT this is not a failure of the executable on IBM, this is just a property of the executable.
This test is intended to check that testing framework do what it supposed to do - in this case catch division by zero. This test failure mean that a compiler does not support this feature.
1) Or regression tests are not used anymore to test *properties* of the platform/compiler/exe 2) Or I include the pragma only for this regression test just for the sake of passing the test.
I definitly prefer option 1 because I consider option 2 as a cheat.
toon
I could add pragma just to show the way how to make test system to work fir this compiler. Though I am not sure it would bring too much value - after all it's users code that needs to include this pragma. Is there any way to manage this with command-line args? Gennadiy.
participants (2)
-
Gennadiy Rozental
-
Toon Knapen