on Sun Aug 05 2007, "Timothy M. Shead" <tshead-AT-k-3d.com> wrote:
David Abrahams wrote:
However, output serialization is only important because both Boost's process_jam_log and ctest/dart currently use a parse-the-build-log strategy to assemble the test results. This approach is hopelessly fragile in my opinion. Instead, each build step in a test should generate an additional target that contains the fragment of XML pertaining to that test, and the complete result should be an XML file that is the catenation of all the fragments. Rene is working on that feature for BBv2. I don't expect it would take more than a couple of days to add this feature to either build system, so the advantage of serialization may be easily neutralized.
There may be some semantics here that I'm missing, but I think ctest is already doing exactly what you describe:
Bill Hoffman (of Kitware) himself told me that parallel builds don't work with ctest because of output interleaving issues. Was he mistaken?
* ctest checks the process return value from each test. A nonzero value equals test failure (unless the test is marked as an expected failure).
* ctest (optionally) parses the test's output (stdout/stderr) using a configurable regular expression. If there is a match, the test fails. The intent is to catch errors reported by some standard logging mechanism, if such a mechanism exists, e.g. messages of the form "CRITICAL ERROR: blah blah blah ...". You can disable this feature if you don't want it.
All of the above is irrelevant to the question at hand, I think.
* ctest produces an XML file that describes the results of each test, including pass/fail, execution time, and test output (stdout/stderr).
A separate XML file for each test? Can it generate these XML files if the tests are run in parallel? That's the key (relevant) issue. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com