Emil Dotchevski wrote:
Then perhaps I didn't understand your proposal. Can you clarify the following:
" a) Testing on the trunk is not as helpful as it should be. My tests results fluctuate all the time as changes in other libraries ripple through to my tests.
Proposal:All tests for a particular library should be run against the latest or next release. "
I read it as you don't want to test against trunk and instead you want to test against the last and the next release.
I envision the following: a) a "release ready" version branch or whatever. This version pass all tests or failing tests are marked up - its "perfect" (lol) b) I make a change in my library, and test against the "release ready" version. This is what I meant by testing against latest or next release. Any failure have to be due to my changes and only my changes. I continue in the loop until it is established that merging my lates "oevre" into the "release ready" version will result in a step forward. c) I ask the release manager for permission to do the merge. Assuming that the "release ready" version is not in a state of flux, he will say yes. d) I'll do the merge and all the boost tests will be run on the "release ready" version - exactly as they are now. e) First we shouldn't have any failures in the library just merged in as that's what I've been testing all along so far. f) Any new failures in any library have to be result of the recent merging of my library. Now its my job to investigate how I broke the interface and/or semantics. It could also be that other libraries are depending on library implemention details or it could be that interface/semantics weren't enforced but are now. in this case a new library test should be written and authors of the dependent libraries need to make adjustments. g) Its also possible that I make the mistake of changing the interface and/or semantics (intentionally or not) without making provision for dependent libraries and applications. The resolution of this situation will be subject to the usual rules for arm wrestling. At the end of this process either the dependent libraries are updated or the merge is backed out of the "release version" There is no test run in the current system which is not run in the proposed system. Instead of carpet bombing approach to testing, its more the "scientific experiment" approach to testing where one variable is changed while others are held constant. Again, the advantages: a) No need/concept for "point release" for fixups. The current "release version" IS the "point release" b) Library authors can contribute their improvements on thier own schedule without having to get in lock step with everyone else. c) Library authors get much more useful information from the tests d) Much less time needs to be consumed to run tests. e) Quarterly releases become relatively simple. f) Library authors don't feel tempted to "cut corners" to make sure their improvements get into the release. Thier improvements will get into the release version as soon as they are ready. g) Users who need the latest bug fixes can download the current "release version" I'm sorry the discussion got somewhat side tracked on this issue of interface/semantic immutability. (OK, it was my fault). Its only somewhat related. I don't think that the fact that some libraries may intentionally modifiy interface/semantics doesn't diminish the utility of this proposal. Robert Ramey