[test] --catch_system_errors questions
Hi ! I'm using Boost.Test for quite a while and now I wonder if the behaviour of "--catch_system_errors" is correct. I've attached a simple test case which just triggers a simple "assert(false);". Running the testcases with trunk bjam yields one failure and two passing tests. See "test.log" for details. Running the testcases against 1.341. yields three failures with _different_ exit codes. See test_1_34_1.log When looking at the output of trunk "with_catch.test", I see --snip--- with_catch: assert_test.cpp:13: void test_assert::test_method(): Assertion `false' failed. <TestLog><TestSuite name="Master Test Suite"><TestCase name="test_assert"><Message file="assert_test.cpp" line="11">one</Message><Info file="assert_test.cpp" line="12">check true passed</Info><TestingTime>0</TestingTime></TestCase></TestSuite></TestLog><TestResult><TestSuite name="Master Test Suite" result="passed" assertions_passed="1" assertions_failed="0" expected_failures="0" test_cases_passed="1" test_cases_failed="0" test_cases_skipped="0" test_cases_aborted="0"></TestSuite></TestResult> EXIT STATUS: 0 --snap-- So, some observations: Obviously, the UTF succeds in catching the "assert". The resulting message is not in the XML-Log The return code is 0 (Success). A quick look at the new docs prototype at http://www.patmedia.net/~rogeeff/html/utf/usage-recommendations/command-line... found "If you got fatal exception somewhere within test case, make program generate coredump by adding extra command line argument " This was not easy to spot since the real test cases succeeded with error code 0 and the message was hidden in the log file. I found this because msvc on windows fires up an annoying debug window instead. So it seems this is not working there. So, my questions/suggestions are: Is this a bug ? Or a feature ? And if its a feature, where can I find the rationale ? I think the caught assertions should lead to a non-zero exit code and a failing test suite. And it would be nice if they got properly XML-formatted (but after a quick scan through the source, this seems difficult...) Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Ingenieurgesellschaft für * voice: ++49 511 262926 57 ! Verkehrs- und Eisenbahnwesen mbH * fax : ++49 511 262926 99 ! Lister Straße 15 * juergen.hunold@ivembh.de ! www.ivembh.de * * Geschäftsführer: ! Sitz des Unternehmens: Hannover * Prof. Dr.-Ing. Thomas Siefer ! Amtsgericht Hannover, HRB 56965 * PD Dr.-Ing. Alfons Radtke !
Juergen Hunold wrote:
Hi !
I'm using Boost.Test for quite a while and now I wonder if the behavior of "--catch_system_errors" is correct.
[...]
When looking at the output of trunk "with_catch.test", I see
--snip--- with_catch: assert_test.cpp:13: void test_assert::test_method(): Assertion `false' failed. <TestLog><TestSuite name="Master Test Suite"><TestCase name="test_assert"><Message file="assert_test.cpp" line="11">one</Message><Info file="assert_test.cpp" line="12">check true passed</Info><TestingTime>0</TestingTime></TestCase></TestSuite></TestLog><TestResult><TestSuite name="Master Test Suite" result="passed" assertions_passed="1" assertions_failed="0" expected_failures="0" test_cases_passed="1" test_cases_failed="0" test_cases_skipped="0" test_cases_aborted="0"></TestSuite></TestResult> EXIT STATUS: 0 --snap--
I've copied the content of your test into unit_test_example_01.cpp. My local run with msvc produces different output (I've added 2>err.xml 1>out.xml). out.xml: <TestLog><TestSuite name="Master Test Suite"><TestCase name="test_assert"><Message file="unit_test_example_01.cpp" line="11">one</Message><Info file="unit_test_example_01.cpp" line="12">check true passed</Info><Exception name="test_assert"> This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. <LastCheckpoint file="unit_test_example_01.cpp" line="12"></LastCheckpoint></Exception><TestingTime>0</TestingTime></TestCase></TestSuite></TestLog> err.xml Assertion failed: false, file unit_test_example_01.cpp, line 13 <TestResult><TestSuite name="Master Test Suite" result="failed" assertions_passed="1" assertions_failed="1" expected_failures="0" test_cases_passed="0" test_cases_failed="1" test_cases_skipped="0" test_cases_aborted="1"></TestSuite></TestResult>
So, some observations: Obviously, the UTF succeds in catching the "assert". The resulting message is not in the XML-Log
It is. You need to split stdout from stderr. Leading message in stderr is unfortunate. I don't believe I can fix it easily
The return code is 0 (Success).
Not in my expirience.
A quick look at the new docs prototype at
http://www.patmedia.net/~rogeeff/html/utf/usage-recommendations/command-line...
found "If you got fatal exception somewhere within test case, make program generate coredump by adding extra command line argument "
This was not easy to spot since the real test cases succeeded with error code 0 and the message was hidden in the log file.
We need to figure out why you have zero result code first.
I found this because msvc on windows fires up an annoying debug window instead. So it seems this is not working there.
So, my questions/suggestions are:
Is this a bug ? Or a feature ? And if its a feature, where can I find the rationale ?
I think the caught assertions should lead to a non-zero exit code and a failing test suite.
And it would be nice if they got properly XML-formatted (but after a quick scan through the source, this seems difficult...)
It seems to be well formed for the most part. Gennadiy
Hi Gennadiy ! An'n Montag 05 Mai 2008 hett Gennadiy Rozental schreven:
Juergen Hunold wrote:
I've copied the content of your test into unit_test_example_01.cpp. My local run with msvc produces different output (I've added 2>err.xml 1>out.xml).
[..]
So, some observations: Obviously, the UTF succeds in catching the "assert". The resulting message is not in the XML-Log
It is. You need to split stdout from stderr. Leading message in stderr is unfortunate. I don't believe I can fix it easily
Oops, my fault :-(( I should have thought about stderr/stdout issues.
The return code is 0 (Success).
Not in my expirience.
Well, using msvc everything works as expected. I'm normally using Linux/gcc and was surprised when my tests ran on Linux and failed using msvc...
We need to figure out why you have zero result code first.
Any ideas where I should start looking ? I've got: gcc -v Using built-in specs. Target: x86_64-suse-linux Configured with: ../configure --enable-threads=posix --prefix=/usr --with-local-prefix=/usr/local --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.2.3 --enable-ssp --disable-libssp --disable-libgcj --with-slibdir=/lib64 --with-system-zlib --enable-shared --enable-__cxa_atexit --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --program-suffix=-4.2 --enable-version-specific-runtime-libs --without-system-libunwind --with-cpu=generic --host=x86_64-suse-linux Thread model: posix gcc version 4.2.3 20071030 (prerelease) (SUSE Linux) and cc-4.3 -v Using built-in specs. Target: x86_64-suse-linux Configured with: ../configure --prefix=/usr --with-local-prefix=/usr/local --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.3 --enable-ssp --disable-libssp --with-bugurl=http://bugs.opensuse.org/ --with-pkgversion='SUSE Linux' --disable-libgcj --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --program-suffix=-4.3 --enable-version-specific-runtime-libs --enable-linux-futex --without-system-libunwind --with-cpu=generic --build=x86_64-suse-linux Thread model: posix gcc version 4.3.1 20080425 (prerelease) [gcc-4_3-branch revision 134659] (SUSE Linux) as well as gcc-4.2 on Kubuntu Hardy returning "Success" for an assert...
I think the caught assertions should lead to a non-zero exit code and a failing test suite.
And it would be nice if they got properly XML-formatted (but after a quick scan through the source, this seems difficult...)
It seems to be well formed for the most part.
Well, forget this one :-)) I think the current output is sufficient for me. It would be nice to add some linebreaks, though. Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Ingenieurgesellschaft für * voice: ++49 511 262926 57 ! Verkehrs- und Eisenbahnwesen mbH * fax : ++49 511 262926 99 ! Lister Straße 15 * juergen.hunold@ivembh.de ! www.ivembh.de * * Geschäftsführer: ! Sitz des Unternehmens: Hannover * Prof. Dr.-Ing. Thomas Siefer ! Amtsgericht Hannover, HRB 56965 * PD Dr.-Ing. Alfons Radtke !
Jürgen Hunold wrote:
Well, using msvc everything works as expected. I'm normally using Linux/gcc and was surprised when my tests ran on Linux and failed using msvc...
We need to figure out why you have zero result code first.
Any ideas where I should start looking ?
I found a bug in *nix signal handling. Fix checked in.
Well, forget this one :-)) I think the current output is sufficient for me. It would be nice to add some linebreaks, though.
It's not intended to be viewed directly. Any number of xml viewers/converters exists to present it in your preferred format. It might be nice to develop some xslt stilesheets to present simple html reports. Anyone interested? Gennadiy
Hi Gennadiy ! An'n Dienstag 06 Mai 2008 hett Gennadiy Rozental schreven:
Jürgen Hunold wrote:
Any ideas where I should start looking ?
I found a bug in *nix signal handling. Fix checked in.
Thanks ! Now things work again. Should this be merged to the release branch for 1.35.1 (if we get one...) ?
Well, forget this one :-)) I think the current output is sufficient for me. It would be nice to add some linebreaks, though.
It's not intended to be viewed directly. Any number of xml viewers/converters exists to present it in your preferred format.
Well, for quick checks I use less and emacs :-))
It might be nice to develop some xslt stilesheets to present simple html reports. Anyone interested?
Certainly, but lacking time (as always ...) Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Ingenieurgesellschaft für * voice: ++49 511 262926 57 ! Verkehrs- und Eisenbahnwesen mbH * fax : ++49 511 262926 99 ! Lister Straße 15 * juergen.hunold@ivembh.de ! www.ivembh.de * * Geschäftsführer: ! Sitz des Unternehmens: Hannover * Prof. Dr.-Ing. Thomas Siefer ! Amtsgericht Hannover, HRB 56965 * PD Dr.-Ing. Alfons Radtke !
Jürgen Hunold wrote:
Hi Gennadiy !
An'n Dienstag 06 Mai 2008 hett Gennadiy Rozental schreven:
Jürgen Hunold wrote:
Any ideas where I should start looking ? I found a bug in *nix signal handling. Fix checked in.
Thanks ! Now things work again. Should this be merged to the release branch for 1.35.1 (if we get one...) ?
Sure. Gennadiy
HI Gennadiy ! On Wednesday 07 May 2008, Gennadiy Rozental wrote:
Jürgen Hunold wrote:
Thanks ! Now things work again. Should this be merged to the release branch for 1.35.1 (if we get one...) ?
Sure.
Will you merge it ? Or shall I do it ? Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Ingenieurgesellschaft für * voice: ++49 511 262926 57 ! Verkehrs- und Eisenbahnwesen mbH * fax : ++49 511 262926 99 ! Lister Straße 15 * juergen.hunold@ivembh.de ! www.ivembh.de * * Geschäftsführer: ! Sitz des Unternehmens: Hannover * Prof. Dr.-Ing. Thomas Siefer ! Amtsgericht Hannover, HRB 56965 * PD Dr.-Ing. Alfons Radtke !
Juergen Hunold <juergen.hunold <at> ivembh.de> writes:
HI Gennadiy !
On Wednesday 07 May 2008, Gennadiy Rozental wrote:
Jürgen Hunold wrote:
Thanks ! Now things work again. Should this be merged to the release branch for 1.35.1 (if we get one...) ?
Sure.
Will you merge it ? Or shall I do it ?
Did we decide on 1.35.1? If we are to go, I'd like the complete trunk state at the time to be released. Gennadiy
Hi, On Tue, May 6, 2008 at 1:00 AM, Gennadiy Rozental <rogeeff@gmail.com> wrote:
Jürgen Hunold wrote:
Well, using msvc everything works as expected. I'm normally using Linux/gcc and was surprised when my tests ran on Linux and failed using msvc...
We need to figure out why you have zero result code first.
Any ideas where I should start looking ?
I found a bug in *nix signal handling. Fix checked in.
Well, forget this one :-)) I think the current output is sufficient for me. It would be nice to add some linebreaks, though.
It's not intended to be viewed directly. Any number of xml viewers/converters exists to present it in your preferred format. It might be nice to develop some xslt stilesheets to present simple html reports. Anyone interested?
I can help with that if You can give me some sample XML files to work on and some hints how the output should look like.
Gennadiy
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
-- ________________ ::matus_chochlik
Matus Chochlik wrote:
It's not intended to be viewed directly. Any number of xml viewers/converters exists to present it in your preferred format. It might be nice to develop some xslt stilesheets to present simple html reports. Anyone interested?
I can help with that if You can give me some sample XML files to work on and some hints how the output should look like.
I don't have any specific position on how it should look like. Probably something that matches some popular format. As for the sample, you can generate one from any test module built with Boost.Test. Both log and report content vary significantly depending on log and report level respectively. Unfortunately there is no schemas available at the moment. Your best shot is to look on XML formatters in xml_log_formatter.ipp and xml_report_formatter.ipp. Gennadiy
Hello Gennadiy, On Wed, May 7, 2008 at 6:53 AM, Gennadiy Rozental <rogeeff@gmail.com> wrote:
Matus Chochlik wrote:
It's not intended to be viewed directly. Any number of xml viewers/converters exists to present it in your preferred format. It might be nice to develop some xslt stilesheets to present simple html reports. Anyone interested?
I can help with that if You can give me some sample XML files to work on and some hints how the output should look like.
I don't have any specific position on how it should look like. Probably something that matches some popular format.
As for the sample, you can generate one from any test module built with Boost.Test. Both log and report content vary significantly depending on log and report level respectively. Unfortunately there is no schemas available at the moment. Your best shot is to look on XML formatters in xml_log_formatter.ipp and xml_report_formatter.ipp.
I have uploaded two xsl templates and a css stylesheet that do the formatting into the vault (folder Home / Miscellaneous, file boost_text_xslt.zip). Hope it helps. Everything worked fine with those logs and reports that I've got but it needs to be tested with more complicated output. Feel free to modify them or post me Your comments if anything needs to be changed and I'll patch it.
Gennadiy
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
-- ________________ ::matus_chochlik
participants (4)
-
Gennadiy Rozental -
Juergen Hunold -
Jürgen Hunold
-
Matus Chochlik