[test] Patches to improve test log detail for better thirdparty integration
In light of recent discussions regarding Boost.Test I've created 3 tickets in trac with patches against trunk that incrementally improve the ease with which Boost.Test can be integrated with thirdparty continuous integration tools, while at the same time improving the output offered. These are patches we've been using for some time locally, the recent discussions highlighted the importance of pushing patches like these to a wider audience. In summary the key changes are: * https://svn.boost.org/trac/boost/ticket/7410 * all test cases/suites have file and line information captured so that external tools can use the Boost.Log output to create links into the source tree, answering questions like, "What is this test case?". It is possible to simply follow the link and see. * https://svn.boost.org/trac/boost/ticket/7417 * update the Boost.Test log output to include summary status information when a test case or suite is completed. This includes the test status, assertions passed and total assertions. This allows live test status to be viewed and nicer tools to be built around this. For example each test can be shown starting, in progress and then complete with the resulting status in detail. I should note that Boost.Test reports currently output this additional test status but this is done only on completion of all tests and so cannot be used to show live progress. * this additional test status as part of the log can be easily captured using regular expressions if HRF is used and many IDEs will allow colourisation, autolinking and so on using similar approaches. * https://svn.boost.org/trac/boost/ticket/7397 * update Boost.Test to use the new Boost.Timer classes introduced in Boost 1.48. This can be suppressed through the use of a macro should the old timers be preferred. The new Boost.Timer provides much better timing information than the old Boost.Timer library. You should refer to the library itself for an explanation of why the previous timer library was deprecated: http://www.boost.org/doc/libs/1_51_0/libs/timer/doc/index.html These patches address what I believe to be important missing features of Boost.Test. We have been using these patches in our systems with Boost 1.49 and enjoy very nice CI integration with Trac Bitten and our Scons build tools. I would therefore encourage those that use Boost.Test and who are interested in these patches in general to take a look. The Boot.Timer patch is important as it also addresses compile issues. See the recent thread here: http://thread.gmane.org/gmane.comp.lib.boost.devel/234411 Gennadiy - assuming you find these acceptable they could be committed to trunk. I don't have commit privileges to do it myself and besides they need reviewed first. Also I would prefer a closer look was taken at the impact they might have on Boost.Test's tests. Finally I should also note that one reason for providing the BOOST_TEST_USE_DEPRECATED_TIMER macro was to circumvent the issue of the new Boost.Timer depending on Boost.System. I don't recall if Boost.System can used header-only? I certainly remember a lot of discussion about it. Jamie
Jamie Allsop <ja11sop <at> yahoo.co.uk> writes:
In light of recent discussions regarding Boost.Test I've created 3 tickets in trac with patches against trunk that incrementally improve the ease with which Boost.Test can be integrated with thirdparty continuous integration tools, while at the same time improving the output offered.
These are patches we've been using for some time locally, the recent discussions highlighted the importance of pushing patches like these to a wider audience.
In summary the key changes are:
* https://svn.boost.org/trac/boost/ticket/7410 * https://svn.boost.org/trac/boost/ticket/7397
These are fine. I'll try to check it in on a weekend
This is not a good direction IMO. This breaks clear separation between test log and report streams. We also do not want duplication of information in two streams. We should consider an approach where report stream is being produced on a fly (in parallel) during test execution. This might be a bit tricky and is not always desirable, but can be enables by special command line switch. Gennadiy
On 25/09/12 18:14, Gennadiy Rozental wrote:
Jamie Allsop<ja11sop<at> yahoo.co.uk> writes:
These are patches we've been using for some time locally, the recent discussions highlighted the importance of pushing patches like these to a wider audience.
In summary the key changes are:
* https://svn.boost.org/trac/boost/ticket/7410 * https://svn.boost.org/trac/boost/ticket/7397
These are fine. I'll try to check it in on a weekend
Excellent thanks!
This is not a good direction IMO. This breaks clear separation between test log and report streams. We also do not want duplication of information in two streams. We should consider an approach where report stream is being produced on a fly (in parallel) during test execution. This might be a bit tricky and is not always desirable, but can be enables by special command line switch.
Interesting idea - having the report be produced on the fly might work well, especially if using human readable format. Essentially you are only a couple of regexes away from useful output. I suppose my main need is to be able to see test progress as: test XXX started... <test now progressing - could take some time> test XXX completed - status - elapsed - assertions passed out of total Having said all that I do not see the harm is adding this to the test log. Here's why. The test log already specifies when a test is started and when a test completes. When a test completes it already gives you the elapsed time. Often you *want* to see the test log output with other messages and so on that you write to the file. Moreover you tend to parse or interpret the test log on the file, tail it or whatever. Therefore simply adding the test status and assertion counts in addition to the time elapsed seems like a good fit to me. I see the role of the report stream as being very different. In the report stream I see the desire to create an arbitrary report in an arbitrary format (like XML). The information contained in the report stream or the log stream is not really the issue. In the log stream you get the logged messages *and* the basic test detail as a simple string so as a human you can read the log and know that test XXX started 'here' output these message and then completed after a period of time. Knowing also whether test XXX passed or failed after that time period and how many assertions were successfully executed is only a slight addition to the current detail. The key point is the log stream is often there to be 'read'. The report stream contains essentially the same test information as the log stream then, except that it *only* contains the test status detail. In essence the report is more like a summary (no log messages and so on - just the test detail). It's primary function is to create a report that can then be used for later interpretation (often by a CI system) - not really for 'reading'. Both streams therefore have a lot of value for different use cases. If anything you could argue that they should expose the same information /precisely/ because they are used for different use cases. For the logging/reading use case why would I get less information? Perhaps controlling the information you get on the log stream could be a command line option that allows people to have that information if they need it, suppressing otherwise. In our use case it is significantly less complicated to parse stdout on the fly to see what is happening. We need to do that anyway for the messages we output. Having to then parse stderr (say) for additional information and merge is more complex. Redirecting the report stream to stdout could help but then I lose the ability to get the test report which I might want separately so send back to a build server. I'll summarise by saying I don't think you compromise the separation between the log and report streams by adding the missing detail to the log stream. If anything you emphasize that the two streams are for different purposes. They both carry the same test information but for different means. Jamie
Gennadiy
_______________________________________________ Unsubscribe& other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
participants (2)
-
Gennadiy Rozental -
Jamie Allsop