on Tue Aug 14 2007, Stefan Seefeld <seefeld-AT-sympatico.ca> wrote:
David Abrahams wrote:
Can you give a brief summary of what QMTest actually does and how Boost might use it?
QMTest is a testing harness.
Meaning, a system for running tests and collecting their results?
Its concepts are captured in python base classes ('Test', 'Suite', 'Resource', 'Target', etc.) which then are implemented to capture domain-specific details. (It is straight forward to customize QMTest by adding new test classes, for example).
What are Resource and Target?
QMTest's central concept is that of a 'test database'. A test database organizes tests. It lets users introspect tests (test types, test arguments, prerequisite resources, previous test results, expectations, etc.), as well as run them (everything or only specific sub-suites, by means of different 'target' implementations
I don't understand what you mean by "run them *by means of* 'target' implementations."
either in serial, or parallel using multi-threading, multiple processes, or even multiple hosts).
Would QMTest be used to drive multi-host testing across the internet (i.e. at different testers' sites), or more likely just within local networks? If the former, how do its facilities for that compare with BuildBot?
Another important point is scalability: While some test suites are simple and small, we also deal with test suites that hold many thousands of tests (QMTest is used for some of the GCC test suites, for example). A test can mean to run a single (local) executable, or require a compilation, an upload of the resulting executable to a target board
Target board?
with subsequent remote execution, or other even more fancy things.
Test results are written to 'result streams' (which can be customized as most of QMTest). There is a 'report' command that merges the results from multiple test runs into a single test report (XML), which can then be translated to whatever output medium is desired.
How could this be useful for boost ?
A good question, but I'm more interested in "how Boost might use it." That is, something like, "We'd set up a server with a test database. QMTest would run on the server and drive testing on each testers' machines, ..." etc.
I found that boost's testing harness lacks robustness.
Our testing system itself seems to be pretty reliable. I think it's the reporting system that lacks robustness.
There is no way to ask seemingly simple questions such as "what tests constitute this test suite ?" or "what revision / date / runtime environment etc. does this result correspond to ?", making it hard to assess the overall performance / quality of the software.
I believe the hardest part is the connection between QMTest and boost.build. Since boost.build doesn't provide the level of introspection QMTest promises, a custom 'boost.build test database' implementation needs some special hooks from the build system. I discussed that quite a bit with Vladimir.
And what came of it? -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com