tools/regression/compiler_status.cpp: possible status bug for compile-only tests
Thought I would try and track down the reason for the "Fail" being shown in the HTML regression report generated by compiler_status. (Is there a better mailing list/forum for reporting problems with these tools?) The following logic is used to determine whether a test passed or failed: xml::element_ptr dbp = xml::parse( file, pth.string() ); const xml::element & db( *dbp ); const xml::element & run_element( find_element( db, "run" ) ); pass = !run_element.name.empty() && attribute_value( run_element, "result" ) != "fail"; So, for a test to be shown as "Pass", !run_element.name.empty() must be true - which requires that the XML test result blob contains a "run" element. Tests which are aimed purely at testing expected compile-time fails (e.g., function/function_30) get the following XML generated by process_jam_log: <test-log library="function" test-name="function_30" test-type="compile" test-program="libs/function/test/function_30.cpp" target-directory="bin/boost/libs/function/test/function_30.test/gcc/debug" toolset="gcc" show-run-output="false"> <compile result="succeed" timestamp="2005-06-17 10:25:10 UTC"> </compile> </test-log> So, the test passed, but compiler_status will show a fail due to the lack of a "run" element. The "pass" assignment should take account of the "test-type". (I can put a patch together, but I might not cover all the cases since I don't know the system too well yet.) phil -- change name before "@" to "phil" for email
On 2005-06-17, Phil Richards <news@derived-software.ltd.uk> wrote:
Thought I would try and track down the reason for the "Fail" being shown in the HTML regression report generated by compiler_status. [...] The "pass" assignment should take account of the "test-type". (I can put a patch together, but I might not cover all the cases since I don't know the system too well yet.)
Not the most error-checked code in the world (should test_type be validated?), but here's a patch that gets rid of a lot of "Fail"s that weren't really fails: Index: tools/regression/compiler_status.cpp =================================================================== RCS file: /cvsroot/boost/boost/tools/regression/compiler_status.cpp,v retrieving revision 1.37 diff -u -r1.37 compiler_status.cpp --- tools/regression/compiler_status.cpp 1 Jun 2005 14:12:47 -0000 1.37 +++ tools/regression/compiler_status.cpp 17 Jun 2005 13:35:30 -0000 @@ -530,6 +530,7 @@ bool do_cell( const string & lib_name, const fs::path & test_dir, + const string & test_type, const string & test_name, const string & toolset, string & target, @@ -563,14 +564,14 @@ xml::element_ptr dbp = xml::parse( file, pth.string() ); const xml::element & db( *dbp ); - const xml::element & run_element( find_element( db, "run" ) ); + const xml::element & test_type_element( find_element( db, test_type ) ); - pass = !run_element.name.empty() - && attribute_value( run_element, "result" ) != "fail"; + pass = !test_type_element.name.empty() + && attribute_value( test_type_element, "result" ) != "fail"; if ( !no_links ) { - note = attribute_value( find_element( db, "run" ), "result" ) == "note"; + note = attribute_value( test_type_element, "result" ) == "note"; // generate bookmarked report of results, and link to it anything_generated @@ -649,7 +650,7 @@ for ( std::vector<string>::const_iterator itr=toolsets.begin(); itr != toolsets.end(); ++itr ) { - anything_to_report |= do_cell( lib_name, test_dir, test_name, *itr, target, + anything_to_report |= do_cell( lib_name, test_dir, test_type, test_name, *itr, target, always_show_run_output ); } phil -- change name before "@" to "phil" for email
On 2005-06-17, Phil Richards <news@derived-software.ltd.uk> wrote:
On 2005-06-17, Phil Richards <news@derived-software.ltd.uk> wrote:
Thought I would try and track down the reason for the "Fail" being shown in the HTML regression report generated by compiler_status. [...] The "pass" assignment should take account of the "test-type". (I can put a patch together, but I might not cover all the cases since I don't know the system too well yet.) Not the most error-checked code in the world (should test_type be validated?), but here's a patch that gets rid of a lot of "Fail"s that weren't really fails: [...]
As I said, not particularly checked, and as it turned out, wrong. This version does seem to work a lot better. Index: tools/regression/compiler_status.cpp =================================================================== RCS file: /cvsroot/boost/boost/tools/regression/compiler_status.cpp,v retrieving revision 1.37 diff -u -r1.37 compiler_status.cpp --- tools/regression/compiler_status.cpp 1 Jun 2005 14:12:47 -0000 1.37 +++ tools/regression/compiler_status.cpp 17 Jun 2005 13:52:47 -0000 @@ -530,6 +530,7 @@ bool do_cell( const string & lib_name, const fs::path & test_dir, + const string & test_type, const string & test_name, const string & toolset, string & target, @@ -563,14 +564,23 @@ xml::element_ptr dbp = xml::parse( file, pth.string() ); const xml::element & db( *dbp ); - const xml::element & run_element( find_element( db, "run" ) ); + std::string test_type_base( test_type ); + if ( test_type_base.size() > 5 ) + { + const string::size_type trailer = test_type_base.size() - 5; + if ( test_type_base.substr( trailer ) == "_fail" ) + { + test_type_base.erase( trailer ); + } + } + const xml::element & test_type_element( find_element( db, test_type_base ) ); - pass = !run_element.name.empty() - && attribute_value( run_element, "result" ) != "fail"; + pass = !test_type_element.name.empty() + && attribute_value( test_type_element, "result" ) != "fail"; if ( !no_links ) { - note = attribute_value( find_element( db, "run" ), "result" ) == "note"; + note = attribute_value( test_type_element, "result" ) == "note"; // generate bookmarked report of results, and link to it anything_generated @@ -649,7 +659,7 @@ for ( std::vector<string>::const_iterator itr=toolsets.begin(); itr != toolsets.end(); ++itr ) { - anything_to_report |= do_cell( lib_name, test_dir, test_name, *itr, target, + anything_to_report |= do_cell( lib_name, test_dir, test_type, test_name, *itr, target, always_show_run_output ); } phil -- change name before "@" to "phil" for email
On 2005-06-17, I wrote: [second attempt at a patch on compiler_status.cpp] Well, it does seem to work quite well, but it highlights a problem with the tests under libs/utility/enable_if - when they get processed by process_jam_log, the test-type extracted ends up being "". Either there is something odd about the Jamfile in there, or process_jam_log has an error. I know nothing about either, so I'm not much help on this one :-) (The reason why it hasn't highlighted before is that the patch *doesn't* default to "run" if the type isn't known, so all tests show as "Fail" even though they haven't.) phil -- change name before "@" to "phil" for email
"Phil Richards" <news@derived-software.ltd.uk> wrote in message news:20050617135816.4AEE99A7BD@derisoft.derived-software.demon.co.uk...
On 2005-06-17, Phil Richards <news@derived-software.ltd.uk> wrote:
On 2005-06-17, Phil Richards <news@derived-software.ltd.uk> wrote:
Thought I would try and track down the reason for the "Fail" being shown in the HTML regression report generated by compiler_status. [...] The "pass" assignment should take account of the "test-type". (I can put a patch together, but I might not cover all the cases since I don't know the system too well yet.) Not the most error-checked code in the world (should test_type be validated?), but here's a patch that gets rid of a lot of "Fail"s that weren't really fails: [...]
As I said, not particularly checked, and as it turned out, wrong. This version does seem to work a lot better...
OK, applied to CVS. Thanks very much for detecting this problem and submitting a patch. In boot-root/tools/regression/test, the existing test programs were renamed to better identify the expected results, and new test programs were added to detect regressions of the problem Phil detected, and also detect similar problems. Those new tests identified an additional minor problem, where compile-fail tests which succeeded were reported as "Warn" rather than "Pass". That has also been fixed. Thanks, --Beman
On 2005-06-17, Phil Richards <news@derived-software.ltd.uk> wrote: [a patch] Much as I feel as though I'm talking to myself in this thread, that's never stopped me before. The current regression log on boost.sourceforge.net shows: gcc 4.0.0 Pass: 84% (123 warnings) Fail: 16% (213) The patch changes this to the more (but not completely) accurate: gcc 4.0.1-pre20050607 Pass: 97% (220 warnings) Fail: 3% (38) Most of the new warnings are actual error output from failed compilations that compiler_status appears to want to link to. I'll take a look at that next... 8 of the fails are due to the lack of test-type information in the test_log.xml file. If nothing else, it makes gcc look a bit healthier... phil -- change name before "@" to "phil" for email
On Fri, Jun 17, 2005 at 04:59:35PM +0100, Phil Richards wrote:
On 2005-06-17, Phil Richards <news@derived-software.ltd.uk> wrote: [a patch]
Much as I feel as though I'm talking to myself in this thread, that's never stopped me before.
I'm listening, FWIW.
If nothing else, it makes gcc look a bit healthier...
That's why I'm listening :-) jon
"Phil Richards" <news@derived-software.ltd.uk> wrote in message news:20050617123730.1CBD743F7@derisoft.derived-software.demon.co.uk...
Thought I would try and track down the reason for the "Fail" being shown in the HTML regression report generated by compiler_status.
(Is there a better mailing list/forum for reporting problems with these tools?)
The following logic is used to determine whether a test passed or failed:
xml::element_ptr dbp = xml::parse( file, pth.string() ); const xml::element & db( *dbp ); const xml::element & run_element( find_element( db, "run" ) ); pass = !run_element.name.empty() && attribute_value( run_element, "result" ) != "fail";
So, for a test to be shown as "Pass", !run_element.name.empty() must be true - which requires that the XML test result blob contains a "run" element. Tests which are aimed purely at testing expected compile-time fails (e.g., function/function_30) get the following XML generated by process_jam_log:
<test-log library="function" test-name="function_30" test-type="compile" test-program="libs/function/test/function_30.cpp"
target-directory="bin/boost/libs/function/test/function_30.test/gcc/debug" toolset="gcc" show-run-output="false"> <compile result="succeed" timestamp="2005-06-17 10:25:10 UTC"> </compile> </test-log>
So, the test passed, but compiler_status will show a fail due to the lack of a "run" element.
The "pass" assignment should take account of the "test-type". (I can put a patch together, but I might not cover all the cases since I don't know the system too well yet.)
I've recently added better test case coverage for compiler_status.cpp, but still no compile-fail test:-( I'll add a case for compile-fail. Assuming it shows the failure you report above, then I'll take a look at your proposed patch. May be a couple of days before I get to it. Thanks for helping, --Beman
"Beman Dawes" <bdawes@acm.org> wrote in message news:d92gnd$sb7$1@sea.gmane.org...
... I've recently added better test case coverage for compiler_status.cpp, but still no compile-fail test:-(
That's totally wrong! There is a compile-fail test, and the results are being reported incorrectly, just as Phil Richards reported. I'll take a look at his patch ASAP, but not tonight. Also I'll look at automating the regression test itself, to avoid future misinterpretations. --Beman
participants (3)
-
Beman Dawes -
Jonathan Wakely -
Phil Richards