Douglas Gregor wrote:
On Jun 4, 2007, at 10:10 AM, Beman Dawes wrote:
The idea is that merge requests would be something as simple as sending an email to some server which is running a script that automatically checks that all criteria are met and then does the merge. [snip] After some more thought, I decided that when a developer requests a test (via the "test on demand" mechanism), it should be possible to request tests on multiple libraries, so that a dependency sub-tree or any portion thereof can be tested.
Rather than building dependency lists into the system (which is a fairly heavyweight approach), it might be simpler to give developers a tool to find which libraries are dependent on their library, and then leave it up to the developer how much or little they test against the dependency tree. If a developer who undertests runs the risk that a merge-into-stable request will fail, because merge-into-stable requests fail if they would cause any other library to fail.
That's three new tools, some of which are non-trivial to develop. All tools are non-trivial to maintain.
Some tools, particularly those that are dependency based, are interesting to speculate about but are not essential. They can be safely ignored for now. And some of the needs may be met by off-the-shelf tools we don't have to develop or maintain. The most critical new (to us) tool would be test-on-demand. I've been very deliberately focusing on figuring out what is needed rather than where we get the tool or how the details work. Now that the need seem fairly firmly defined, we can start looking at what tools are available to meet those needs.
We clearly need tools to improve the Boost development and release process. The problem is that while good tools can help the process, poor tools can hurt us even more than no tools. We can't build new tools until we've fixed or replaced the existing tools, and we can't build new tools without a solid plan for maintaining those tools.
I'm tired of waiting. For what I'm proposing, bjam is good enough as it stands now. The downstream reporting system is orthogonal to test-on-demand.
Look at the 1.34 release series... the thing that's been holding us back most of all is that the testing and test reporting tools are broken. 1.34.1 is stalled because we have failures on one platform, but nobody can see what those failures actually are: the test reporting system removed all of the important information.
I agree with most of Beman's write-up, but it pre-supposes a robust testing system for Boost that just doesn't exist.
That may be true for the whole system testing and reporting release managers care about, but for for a developer wanting to test a single library bjam works pretty well, and I suspect it will work well for tests on a small number of dependent libraries too. But regardless, the test-on-demand system should be independent of how the testing is actually run. If we change to a different build system or test execution framework, it would nice if the procedures as seen by a developer don't change much.
I hypothesize that the vast majority of the problems with our release process would go away without a single change to our process, if only we had a robust testing system.
I think you have to change the process at least enough so that a stable branch is always maintained and developers can test their library's development branch on deman against the stable branch. The current "wild-west" problems in the trunk would not go away just because the testing system worked better, was more responsive, etc.
We have only so much volunteer time we can spend. At this point, I think our best bet is to spend it making the regression testing infrastructure work well; then we can move on to a new process with its new tools.
I hope you aren't counting Subversion as a "new" tool! And what about starting the next release from the last release, rather than branching the development truck? That is a new process, but it is one we can start right away. In general, I'd like to increment into new processes as we get them figured out, and new tools when we find something that will better support our processes. --Beman