on Fri Aug 17 2007, Stefan Seefeld <seefeld-AT-sympatico.ca> wrote:
David Abrahams wrote:
So QMTest doesn't schedule build/test processes? It's just a database manager?
Yes it does 'schedule' them, but according to rather specific requirements. You wouldn't tell QMTest to run a test suite nightly, for example, or hook up to some other external triggers (checkins, say). Rather, you specify what (sub-) testsuite to run, and QMTest will work out the order etc.
The order in which to run tests? I don't believe we have any dependency relationships (at least, not encoded ones) that can help QMTest.
> How could this be useful for boost ? A good question, but I'm more interested in "how Boost might use it." That is, something like, "We'd set up a server with a test database. QMTest would run on the server and drive testing on each testers' machines, ..." etc. Still looking for that. Yes, I realize that. But as I indicated earlier, I'm not convinced QMTest is a good tool to schedule / drive that.
I meant I want some kind of analogous statement about a way we could use it that you *are* convinced of.
* handle all test suite runs through QMTest
too vague.
* aggregate test results in a central place using QMTest,
So QMTest stores results, OK.
and manage interpretation (including expectations)
How would one do that?
to generate test reports.
Does QMTest generate reports?
There's no a priori reason that Boost.Build needs to maintain the test database, is there? No.
So what are the alternatives to that arrangement?
A boost-specific test database implementation could work like this:
* by default, map each source file under libs/*/test/ to a test id, and provide a default test type which 1) compiles, 2) runs the result, 3) checks exit status and output. Default compile and link options could be generated per component (boost library).
* For cases where the above doesn't work, some special test implementation can be used (that incorporate the special rules now part of the various Jamfiles).
I think I understand. Essentially, one would need to implement Python classes whose instances represent each test and know how to do the testing. One could generate Jamfiles for the difficult cases. But how would we represent the tests? Python code? An actual database? As I see it right now, the most significant benefit available from QMTest is in the fact that it robustly controls the running of each test, capturing its results, and comparing those results with what's expected. Is that right? -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com