"Jeff Garland" <jeff@crystalclearsoftware.com> writes:
On Sat, 19 Mar 2005 20:12:55 -0500, David Abrahams wrote
"Jeff Garland" <jeff@crystalclearsoftware.com> writes:
Fair enough -- I was almost afraid to ask since I hate to make the release process any harder than it is now.
Agreed, it's too hard, but that shouldn't stop us from talking about what we would be doing in an ideal world. Accordingly:
- A health report for the latest release should always be available on the website.
Yes, it is there -- in the middle of the front page, but even so I missed it. So I think the link belongs under 'regression tests' called something like 'current release'.
I think it should also be available from an obvious link when you go to get information on or download the current release.
- Regressions from the previous release are nice to know but less important. I realize we show both in one report, but this may help us adjust our emphasis or coloring (maybe it's already perfect in the user report; I don't know)
In fact, I think from the user perspective the question really goes something like: "I'm using Intel 8.1 on Windows and Linux and I want to use Python, smart_ptr, and serialization -- can I expect these libraries to work with release xyz?" And several variations on that theme. So in an "ideal world" scenario I would have a web form where the user could enter her needs and a script would filter the regression results down to the set of interest for the user.
Okay, yeah; that would be an improvement.
- A health report for the current state of the repository should always be available on the website.
- Regressions from the previous release are crucial to know also
- When we branch for a release, we absolutely must track the release branch, but we also should be continuing to display the health of the trunk
Agree -- I think the big blocker here is expanding the set of regression testers during this period.
This is part of why I think BuildBot is a good idea.
Another factor is that the regression compilers/platforms tested between releases is not really stable. It has been growing, so we have now have 'new results' that can't really 'diff' from the last release. For example, we now have a regression tester for Solaris which we didn't have in 1.32. I'm not sure that's obvious from the way we display results.
It probably isn't.
- We ought to have a system for automatically notifying anyone who checks in a regression, and displaying information about the change responsible for the regression on the status page.
Do we even have a way of tracking the check-ins?
CVS?
That might be a good first step. I notice that sourceforge seems to be generating some sort of email when I check-in, but I don't know of a way to subscribe to the changelist.
We can set up a mailing list for it to send to, if you want to see those. But I don't think that would solve the problem by itself.
- There should be a way for a developer to request testing of a particular branch/set of revisions
I'd put this high on my list. Without it there is no practical way for developers to regression test on a branch. Which means that using branches for development isn't really that practical.
- There should be enough computing power to handle all these tests in a timely fashion.
Guess it depends on what you consider timely -- 1 minute, 1 hour, 1 day? We are somewhere in the 2 day sort of range now. From the developer perspective, the ideal world would be 'right now'.
Yeah, I mean something very close to "right now."
I've got these changes I'm working on, I've tested on my core compilers and I'm ready to see the results for other compilers. It seems like most testers run one regression test per day, while others run several. So depending on when you check-in, it takes up to a couple days to really see the results of all platforms/compilers. The only way I see us getting closer to the ideal is more machines really dedicated to just Boost testing...
Those irons are in the fire now; see my "Testing Farm" thread.
We also need to discuss how the main trunk will be treated. Gennadiy has suggested in the past that checking in breaking changes to the trunk is a perfectly legitimate technique for test-driven development. I agree in principle, but that idea seems to generate a lot of friction with other developers trying to stabilize their test results.
I agree with him about wanting to use the compiler to find breakage, but the problem is that his particular library is one that many libraries depend on. As a result, it really needs to stay stable during the release period to ensure that we don't have 2 days of downtime while something Boost-wide is broken.
If we could initiate tests on a branch by request we wouldn't have this problem; he could run all those tests before merging.
So I really think that we need to start thinking about a dependency analysis of Boost and an added freeze date for 'core libraries' that need to stay stable during the release process.
I'd really like to avoid that.
Developers will have to finish what they want in the release earlier. This could certainly be relaxed if branch-testing is available since a developer could be much more sure of avoiding mainline breakage...
Yup.
The ability to request testing of a branch might go a long way toward eliminating that sort of problem.
Agree completely.
-- Dave Abrahams Boost Consulting www.boost-consulting.com