This is part 1 of my analysis of our prospects for improving the Boost release process by improving the tools. I'll post something based on this to the wiki after y'all give me an earful about what I got wrong or overlooked. Build System ============ I'm only considering BBv2 and cmake here, because we currently use the former and Doug and Troy have been putting lots of effort into the latter, and Kitware has volunteered to support it for us. Cmake has lots of advantages over BBv2 for developers (e.g. instant build startup time, use favorite development environment) and packagers (e.g. one-line installer target declaration), but these factors are more important for day-to-day development than they are for testing. Instant build startup time would improve test turnaround time, of course, but build startup does not dominate our testing time yet. BBv2 also has overall speed/memory-consumption problems. These might be addressable using the Boehm GC, but the last attempt resulted in crashes (more investigation needed). BBv2 has a few advantages over Cmake for testing. 1. BBv2 can build with multiple toolsets and/or variants in one invocation. That, however, is trivially overcome by using a wrapper script over cmake. 2. BBv2 serializes the output of simultaneous tests, which makes it possible to run tests in parallel with -jN. This is really important for test turnaround time. Even uniprocessors benefit from a low N (like 2) IME, due to disk/cpu tradeoffs. The IBM testers are currently using -j16 to great effect on their big iron. However, output serialization is only important because both Boost's process_jam_log and ctest/dart currently use a parse-the-build-log strategy to assemble the test results. This approach is hopelessly fragile in my opinion. Instead, each build step in a test should generate an additional target that contains the fragment of XML pertaining to that test, and the complete result should be an XML file that is the catenation of all the fragments. Rene is working on that feature for BBv2. I don't expect it would take more than a couple of days to add this feature to either build system, so the advantage of serialization may be easily neutralized. 3. Platform/compiler-independent build specification. This one seems pretty important on the face of it. If library authors' tests won't run on platforms to which they don't have direct access, we'll need to find people to port and maintain test specifications for the other platforms. On the other hand, very few tests are doing anything fancy when it comes to build configuration. To understand the real impact of this BBv2 feature, someone would have to objectively analyze the cmake build specifications Doug and Troy have been working on for complexity and portability. If either of those guys can be objective, it would be fastest if they'd do the analysis. 4. Cmake's built-in programming language is even more horrible, IMO, than bjam's. I bet you get used to it, like anything else, but it does present a barrier to entry. On the other hand, it's my impression that most programming jobs can be done more simply and directly under Cmake because it lacks BBv2's multiple-toolset/variant builds, and so has no need for virtual targets. Neither systems can trace #include dependencies through macros (e.g. #include SOME_MACRO(xyz)), which occasionally leads to inaccurate results from incremental testing. This is an important problem that should be fixed. Based on the above points, I'd say either build system has the potential to work for Boost's testing needs and neither one is substantially closer to being ideal for us (for testing). Since our systems use BBv2 now, it makes sense to concentrate effort there. However, if cmake were to suddenly acquire the XML fragment generation and the analysis of point 3 above showed that portable build specification is easily accomplished in cmake or not very important to our testing needs, I'd probably be vote the other way. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com