I should note that the serialization library changes only very infrequently - maybe once every few weeks. I would question re-runing the serialization library tests just because something that the serialization library depends upon changes. Certainly this doesn't do me any good. If someone breaks/changes and api that the serialization library depends upon, does that person even find out about it? Much less track it down? Not that this is a big deal as it almost never happens. The last time it happened that I remember was when all the serialization tests for comeau started to fail when something in Boost Test changed. That was 6 months ago and nothing has changed. A main cause of this problem bjam dependency analysis re-runs all tests on Library X even if library X hasn't changed. But boost has other scalability problems. The testing/release regimen permits libraries to be more tightly coupled than they probably should be. Ideally I would like to see something like: a) Each maintainer has his own branch. b) He makes changes on his own machine c) He tests his changes on his own machine. b) He makes changes on his branch. c) Whenever he makes a change, tests are run on that branch for that library. d) When he is statisfied with c) above, he merges changes from his branch into the main trunk. e) Whenever d) occurs, tests are run on the main branch for any libraries newly merged in. f) The health of the main trunk "should" be very good. Periodically, and much more frequently than now, (maybe every 60 days or so), Exhaustive tests are run on the trunk - just as they are now. Any problems should be small. These are fixed and a new release is made - zip an tarbal. This would a) diminish requirements for testing immensly. b) testing resources would not be wasted on repeating the same tasks. c) library authors could not rely on other libraries to test their code. I know no one does this intentionally, but we're all human, and w're under pressure and writing tests IS a pain and, etc.... d) tests would by necessity have to be more complete and focused. Presumable these would build incrementally. e) result in much more frequent new releases. If a bug is found, it would be practical to just wait for the next release to fix it rather then trying to "stop the presses" f)Errors would be easier to find. Since we're all testing our changes agains the trunk rather than everyone elses changes any error during development has to be library writers. An error that shows up when changes are merged to the trunk has to be due to an API change in another library. g) Authors wouldn't be under pressure to get feature Y crammed in to make the next release date. Each would work at his how pace knowing that shortly after its ready, it will appear in the official release. h) Author's wouldn't have to hold back changes that they think are OK because they just might throw a monkey wrench into the whole works. It seems to me that this should be the main benefit of the source control system and that we are not receiving that benefit. Robert Ramey Rene Rivera wrote:
David Abrahams wrote:
"troy d. straszheim" <troy@resophonic.com> writes:
Rene Rivera wrote:
One thing I should make clear about my suggestion to reduce the combinations... Is that it doesn't mean reducing the number of types that get serialized, rather increasing those and having more complex and comprehensive test.
I need to think about it more... for now, I carpet bomb. But it clearly would get out of control as types, archives, platforms and versions proliferate. No question.
It sounds like, from what Rene has said, it already is out of control.
I did another incremental test run today which spent most of the time retesting serialization, probably some minor change. But it ran from 10:30am to 8:00pm, 11.5 hours. So yes, I consider that out of control.
Anything that makes it very difficult and/or expensive for a tester to complete a testing run needs to be fixed rather urgently. I hope you realize that, and I think you probably do, but from your posting it wasn't clear.
I hope everyone realizes this.