On Jun 4, 2007, at 10:10 AM, Beman Dawes wrote:
The idea is that merge requests would be something as simple as sending an email to some server which is running a script that automatically checks that all criteria are met and then does the merge. [snip] After some more thought, I decided that when a developer requests a test (via the "test on demand" mechanism), it should be possible to request tests on multiple libraries, so that a dependency sub-tree or any portion thereof can be tested.
Rather than building dependency lists into the system (which is a fairly heavyweight approach), it might be simpler to give developers a tool to find which libraries are dependent on their library, and then leave it up to the developer how much or little they test against the dependency tree. If a developer who undertests runs the risk that a merge-into-stable request will fail, because merge-into-stable requests fail if they would cause any other library to fail.
That's three new tools, some of which are non-trivial to develop. All tools are non-trivial to maintain. We clearly need tools to improve the Boost development and release process. The problem is that while good tools can help the process, poor tools can hurt us even more than no tools. We can't build new tools until we've fixed or replaced the existing tools, and we can't build new tools without a solid plan for maintaining those tools. Look at the 1.34 release series... the thing that's been holding us back most of all is that the testing and test reporting tools are broken. 1.34.1 is stalled because we have failures on one platform, but nobody can see what those failures actually are: the test reporting system removed all of the important information. I agree with most of Beman's write-up, but it pre-supposes a robust testing system for Boost that just doesn't exist. I hypothesize that the vast majority of the problems with our release process would go away without a single change to our process, if only we had a robust testing system. We have only so much volunteer time we can spend. At this point, I think our best bet is to spend it making the regression testing infrastructure work well; then we can move on to a new process with its new tools. - Doug