question regarding assert and lightweight test
It seems to me that when I get an assertion in my GCC compilers the test is still being marked as "pass" I'm using lightweight test in the follow way. Does anyone have any suggestions? Robert Ramey int main(int argc, char * argv[]){ boost::serialization::global_lock::get_mutable_instance().lock(); BOOST_TRY{ test_main(argc, argv); } #ifndef BOOST_NO_EXCEPTION_STD_NAMESPACE BOOST_CATCH(const std::exception e){ BOOST_TEST(e.what()); } #endif BOOST_CATCH(...){ BOOST_TEST("failed with uncaught exception:"); } BOOST_CATCH_END boost::serialization::global_lock::get_mutable_instance().unlock(); return boost::report_errors(); }
Robert Ramey:
It seems to me that when I get an assertion in my GCC compilers the test is still being marked as "pass"
I'm using lightweight test in the follow way. Does anyone have any suggestions? ...
BOOST_TEST(e.what());
...
BOOST_TEST("failed with uncaught exception:");
BOOST_TEST( expression ) tests whether expression is true. You might need BOOST_ERROR for the above two if you want them to be flagged as a test failure and output their argument as the reason. I took a quick look at the serialization tests though and I don't see anything resembling the above; everything in test_tools.hpp seems fine.
Peter Dimov wrote:
Robert Ramey:
It seems to me that when I get an assertion in my GCC compilers the test is still being marked as "pass"
I'm using lightweight test in the follow way. Does anyone have any suggestions? ...
BOOST_TEST(e.what());
...
BOOST_TEST("failed with uncaught exception:");
BOOST_TEST( expression ) tests whether expression is true. You might need BOOST_ERROR for the above two if you want them to be flagged as a test failure and output their argument as the reason.
I took a quick look at the serialization tests though and I don't see anything resembling the above; everything in test_tools.hpp seems fine.
I'm doing this on a branch "serialization next release" rather than trunk. That might explain it. Robert Ramey
I've traced through the code with the gdb and seems to me that when assert is invoked, the program bails more or less immediatly without jumping back to the orginal main or invoking an exception that can be caught. This results in tests which provoke assertions as being marked "passed". Actually, this is unbelievable to me and I have to believe that I'm missing something really dumb. I also took a cursory look at Boost Test code and don't see any special provision made for assert. Also I investigated BOOST_ASSERT. It looks to me that this code is not conditional upon NDEBUG not being defined - that is that the code included by BOOST_ASSERT would still appear in a release build. Again, I can't help but feel that I'm missing something really obvious. Robert Ramey Robert Ramey wrote:
Peter Dimov wrote:
Robert Ramey:
It seems to me that when I get an assertion in my GCC compilers the test is still being marked as "pass"
I'm using lightweight test in the follow way. Does anyone have any suggestions? ...
BOOST_TEST(e.what());
...
BOOST_TEST("failed with uncaught exception:");
BOOST_TEST( expression ) tests whether expression is true. You might need BOOST_ERROR for the above two if you want them to be flagged as a test failure and output their argument as the reason.
I took a quick look at the serialization tests though and I don't see anything resembling the above; everything in test_tools.hpp seems fine.
I'm doing this on a branch "serialization next release" rather than trunk. That might explain it.
Robert Ramey
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey:
I've traced through the code with the gdb and seems to me that when assert is invoked, the program bails more or less immediatly without jumping back to the orginal main or invoking an exception that can be caught. This results in tests which provoke assertions as being marked "passed".
assert should in principle call abort(), which should return a failure exit code to the OS, so the test should fail...
Actually, this is unbelievable to me and I have to believe that I'm missing something really dumb. I also took a cursory look at Boost Test code and don't see any special provision made for assert.
Also I investigated BOOST_ASSERT. It looks to me that this code is not conditional upon NDEBUG not being defined - that is that the code included by BOOST_ASSERT would still appear in a release build. Again, I can't help but feel that I'm missing something really obvious.
BOOST_ASSERT works in three modes. By default, it maps to assert, and assert does obey NDEBUG. If BOOST_DISABLE_ASSERTS is defined, BOOST_ASSERT is compiled out. If BOOST_ENABLE_ASSERT_HANDLER is defined, failed asserts call a function that can be supplied by the user. So if your code uses BOOST_ASSERT, you can define a custom handler in your tests that prints a message and cause the test to fail. http://www.boost.org/libs/utility/assert.html
Well, that makes a lot of sense. But when I build/test with bjam the tail of bjam.log looks like: gcc.archive ..\..\..\bin.v2\libs\serialization\build\gcc-3.4.4\debug\link-static \libboost_serialization-gcc34-d-1_34_1.a gcc.link ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_132_text_archiv e.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archive.exe testing.capture-output ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_1 32_text_archive.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archiv e.run ====== BEGIN OUTPUT ====== assertion "result.second" failed: file "..\..\..\libs\serialization\src\extended _type_info.cpp", line 75 EXIT STATUS: 34304 ====== END OUTPUT ====== **passed** ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_132_text_arch ive.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archive.test ...updated 32 targets... Peter Dimov wrote:
Robert Ramey:
I've traced through the code with the gdb and seems to me that when assert is invoked, the program bails more or less immediatly without jumping back to the orginal main or invoking an exception that can be caught. This results in tests which provoke assertions as being marked "passed".
assert should in principle call abort(), which should return a failure exit code to the OS, so the test should fail...
Actually, this is unbelievable to me and I have to believe that I'm missing something really dumb. I also took a cursory look at Boost Test code and don't see any special provision made for assert.
Also I investigated BOOST_ASSERT. It looks to me that this code is not conditional upon NDEBUG not being defined - that is that the code included by BOOST_ASSERT would still appear in a release build. Again, I can't help but feel that I'm missing something really obvious.
BOOST_ASSERT works in three modes. By default, it maps to assert, and assert does obey NDEBUG. If BOOST_DISABLE_ASSERTS is defined, BOOST_ASSERT is compiled out. If BOOST_ENABLE_ASSERT_HANDLER is defined, failed asserts call a function that can be supplied by the user. So if your code uses BOOST_ASSERT, you can define a custom handler in your tests that prints a message and cause the test to fail.
http://www.boost.org/libs/utility/assert.html
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
on Fri Nov 02 2007, "Robert Ramey" <ramey-AT-rrsd.com> wrote:
Well, that makes a lot of sense. But when I build/test with bjam the tail of bjam.log looks like:
gcc.archive ..\..\..\bin.v2\libs\serialization\build\gcc-3.4.4\debug\link-static \libboost_serialization-gcc34-d-1_34_1.a gcc.link ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_132_text_archiv e.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archive.exe testing.capture-output ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_1 32_text_archive.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archiv e.run ====== BEGIN OUTPUT ====== assertion "result.second" failed: file "..\..\..\libs\serialization\src\extended _type_info.cpp", line 75
EXIT STATUS: 34304 ====== END OUTPUT ====== **passed** ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_132_text_arch ive.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archive.test ...updated 32 targets...
This looks like a serious bug, assuming you weren't doing a parallel build (in which case the **passed** message might belong to a different test). Are you using the regular windows command shell to build with a cygwin GCC? -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
David Abrahams wrote:
on Fri Nov 02 2007, "Robert Ramey" <ramey-AT-rrsd.com> wrote:
Well, that makes a lot of sense. But when I build/test with bjam the tail of bjam.log looks like:
gcc.archive ..\..\..\bin.v2\libs\serialization\build\gcc-3.4.4\debug\link-static \libboost_serialization-gcc34-d-1_34_1.a gcc.link ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_132_text_archiv e.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archive.exe testing.capture-output ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_1 32_text_archive.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archiv e.run ====== BEGIN OUTPUT ====== assertion "result.second" failed: file "..\..\..\libs\serialization\src\extended _type_info.cpp", line 75
EXIT STATUS: 34304 ====== END OUTPUT ====== **passed** ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_132_text_arch ive.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archive.test ...updated 32 targets...
This looks like a serious bug, assuming you weren't doing a parallel build (in which case the **passed** message might belong to a different test).
I'm not using a parallel build.
Are you using the regular windows command shell to build with a cygwin GCC?
However, my default shell is the MKS Korn shell. Given that no one else has reported this that might be an issue. I don't know how bjam works but I would have thought that it invoked the compiler directly rather than through the current shell. I can try the the normal DOS-like shell or the cygwin shell if you think that would make a difference. Robert Ramey
on Fri Nov 02 2007, "Robert Ramey" <ramey-AT-rrsd.com> wrote:
David Abrahams wrote:
on Fri Nov 02 2007, "Robert Ramey" <ramey-AT-rrsd.com> wrote:
Well, that makes a lot of sense. But when I build/test with bjam the tail of bjam.log looks like:
gcc.archive ..\..\..\bin.v2\libs\serialization\build\gcc-3.4.4\debug\link-static \libboost_serialization-gcc34-d-1_34_1.a gcc.link ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_132_text_archiv e.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archive.exe testing.capture-output ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_1 32_text_archive.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archiv e.run ====== BEGIN OUTPUT ====== assertion "result.second" failed: file "..\..\..\libs\serialization\src\extended _type_info.cpp", line 75
EXIT STATUS: 34304 ====== END OUTPUT ====== **passed** ..\..\..\bin.v2\libs\serialization\test\test_shared_ptr_132_text_arch ive.test\gcc-3.4.4\debug\link-static\test_shared_ptr_132_text_archive.test ...updated 32 targets...
This looks like a serious bug, assuming you weren't doing a parallel build (in which case the **passed** message might belong to a different test).
I'm not using a parallel build.
Are you using the regular windows command shell to build with a cygwin GCC?
However, my default shell is the MKS Korn shell.
Oh, forget it. That's unsupported. Use cmd, please.
Given that no one else has reported this that might be an issue. I don't know how bjam works but I would have thought that it invoked the compiler directly rather than through the current shell.
No, it uses the shell, as you can clearly see in the error messages for failed tests, that are all setting shell variables using redirection, and chaining with && and ||.
I can try the the normal DOS-like shell or the cygwin shell if you think that would make a difference.
If you use a "windows bjam" (built from the cmd shell) you need to run it from inside cmd. If you use a "cygwin bjam" (built from cygwin bash) you need to run it from inside cygwin bash. Those are the only two options on windows, and Boost.Build doesn't support cygwin at all anymore, so that may cease to work any day now. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
David Abrahams wrote:
I can try the the normal DOS-like shell or the cygwin shell if you think that would make a difference.
If you use a "windows bjam" (built from the cmd shell) you need to run it from inside cmd.
I run the Windows build of bjam under cygwin all the time: and for that matter to build and test my stuff under cygwin. You just have to remember that when bjam shell's out it'll be using cmd.exe to execute commands and not cygwin. John.
I ran it from inside cmd.exe and got the same result. John Maddock wrote:
David Abrahams wrote:
I can try the the normal DOS-like shell or the cygwin shell if you think that would make a difference.
If you use a "windows bjam" (built from the cmd shell) you need to run it from inside cmd.
hmmm ... yet another hidden variable. I built bjam from the tools/build directory - If I remember correctly and this was some time ago so that's another possible source.
I run the Windows build of bjam under cygwin all the time: and for that matter to build and test my stuff under cygwin. You just have to remember that when bjam shell's out it'll be using cmd.exe to execute commands and not cygwin.
Do you visually inspect the bjam.log when you do this? Do you know for a fact that an "assert" wiill result in a "failed" test? If this behavior were occuring in your environmemnt, how would you know about it? I only discovered it when I found a couple of tests which only "failed" in release mode - which is never tested by default. Only when I went to investigate this did I discover that the there was a failed assertion that was in fact being marked "passed" Is there a test in the bjam test suite which verifys this? Is this test being run on all platforms whenever bjam is changed? robert Ramey
John.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
I ran it from inside cmd.exe and got the same result.
John Maddock wrote:
David Abrahams wrote:
I can try the the normal DOS-like shell or the cygwin shell if you think that would make a difference.
If you use a "windows bjam" (built from the cmd shell) you need to run it from inside cmd.
hmmm ... yet another hidden variable.
I built bjam from the tools/build directory - If I remember correctly and this was some time ago so that's another possible source.
I run the Windows build of bjam under cygwin all the time: and for that matter to build and test my stuff under cygwin. You just have to remember that when bjam shell's out it'll be using cmd.exe to execute commands and not cygwin.
Do you visually inspect the bjam.log when you do this? Do you know for a fact that an "assert" wiill result in a "failed" test? If this behavior were occuring in your environmemnt, how would you know about it?
I only discovered it when I found a couple of tests which only "failed" in release mode - which is never tested by default. Only when I went to investigate this did I discover that the there was a failed assertion that was in fact being marked "passed"
Is there a test in the bjam test suite which verifys this?
No, because it's not bjam thing. The Boost.Build testsuite does include test/regression.py test that tests that non-zero return from a program is detected. It worked on windows a couple of weeks ago, and nothing was changed in that department since then. Can you reproduce this problem using a minimal testcase -- preferrably not using any of boost.serialization, or any non-standard assert functionality?
Is this test being run on all platforms whenever bjam is changed?
No. I plan to change regression tests to run those, however. - Volodya
Vladimir Prus wrote:
No, because it's not bjam thing. The Boost.Build testsuite does include test/regression.py test that tests that non-zero return from a program is detected. It worked on windows a couple of weeks ago, and nothing was changed in that department since then. Can you reproduce this problem using a minimal testcase -- preferrably not using any of boost.serialization, or any non-standard assert functionality?
Hmmm - I would be happy to do this with something like: #include <cassert> int main(int argc, char * argv[]){ assert(false); } I thought it would be easy to name it assert-fail.cpp add to the directory which contains compile-fail.cpp etc. and add assert-fail to the Jamfile.v2 and rerun the test. But I found no such Jamfile.v2. So I'm not sure what to do with this. It's not at all clear to me how Boost.Build (as used for boost testing) is beiing tested aside from just running it on the trunk all the time. Robert Ramey
"Robert Ramey" <ramey@rrsd.com> wrote in message news:fgferc$i67$1@ger.gmane.org...
Actually, this is unbelievable to me and I have to believe that I'm missing something really dumb. I also took a cursory look at Boost Test code and don't see any special provision made for assert.
Boost.Test does have a provision for assert where possible. There is a test to this extend as well: prg_exec_fail3. Gennadiy
participants (6)
-
David Abrahams -
Gennadiy Rozental -
John Maddock -
Peter Dimov -
Robert Ramey -
Vladimir Prus