Independent Libraries Versioning proposal
Hi, Following David A. recommendation I put up a wiki page that presents initial version of my proposal for boost development environment, testing and release procedures. http://svn.boost.org/trac/boost/wiki/IndependentLibraryVersioning It's still missing one key piece - version selection algorithm. I just don't have time to complete it now and won't until weekend. But it should be complete enough to explain all the key points (and some more). I shamelessly stole couple statements from Beman's objective section. Well, we do have similar objectives ;) Any comments are welcome. Regards, Gennadiy
Gennadiy Rozental wrote:
Hi,
Following David A. recommendation I put up a wiki page that presents initial version of my proposal for boost development environment, testing and release procedures.
http://svn.boost.org/trac/boost/wiki/IndependentLibraryVersioning
It's still missing one key piece - version selection algorithm. I just don't have time to complete it now and won't until weekend. But it should be complete enough to explain all the key points (and some more). I shamelessly stole couple statements from Beman's objective section. Well, we do have similar objectives ;)
Any comments are welcome.
I like the idea of allowing individual libraries and their dependencies being released between full Boost versions when deemed necessary. However, from an end user's point of view, allowing a version of some library to be released with an earlier dependency version than a version of another library with a later dependency version, where the first library no longer works with the later dependency version, will be a nightmare. Because of that I believe that any individual library needs to be co-ordinated with its dependencies so that changes in a dependency library, which may be released on its own or with another library, does not break the original individual library. When I say "break" the original individual library I am not speaking about a bug which may occur in a dependency but rather a change to the public ( or protected ) programming interface of the dependent, which I will call, perhaps erroneously, the ABI. The reason I think such a situation will be a nightmare to end-users is because an end-user may then have more than one version of the same dependency library within the structure of using a general Boost release, and pointing to a different set of header files, with possibly the equivalent libraries files for non-header-only libraries, for each different use of the libraries, will create great logistical difficulties to the smooth use of Boost. As an example, suppose I use library A and library B, both of which have library C as a dependency. Library A uses version 1 of library C and library B uses version 2 of library C. Library A is broken, ABI-wise, with version 2 of library C. Now I want to use both library A and library B in my program ( or module ), often from within the same source file compilation. How in the world do I have library A pointing to its own version of library C's header files while library B points to its own version of library C's header files ? This can be complicated further if library C has lib files which either statically are added to my module or are linked to dynamically through a shared library. This situation just increases the nightmare of trying to use both the libraries. As I suggested in another post on this thread, I think it is certainly possible for an individual library to be released between full Boost releases, but only if the Boost developer of that library makes sure that his library is always co-ordinated ABI-wise with any of its dependencies. Likewise if a dependency of a library is going to be released itself, it is imperative that any libraries for which that library is a dependency, and for which the change to the dependency will create an ABI problem, also be released with an update at the same time which fixes the ABI problem. It is nice to think that any Boost library should be able to be released at any time to work with any version of a dependency, but the reality of such a plan will often lead to chaos from the end-user's point of view. I am not saying this can not be done for unique individual situations, since I have done it myself using the Boost regex++ library, but it needs very careful consideration so as not to create a nightmare for the end-user.
"Edward Diener" <eldiener@tropicsoft.com> wrote in message news:f496rp$she$1@sea.gmane.org...
Gennadiy Rozental wrote:
Hi,
Following David A. recommendation I put up a wiki page that presents initial version of my proposal for boost development environment, testing and release procedures.
http://svn.boost.org/trac/boost/wiki/IndependentLibraryVersioning
It's still missing one key piece - version selection algorithm. I just don't have time to complete it now and won't until weekend. But it should be complete enough to explain all the key points (and some more). I shamelessly stole couple statements from Beman's objective section. Well, we do have similar objectives ;)
Any comments are welcome.
I like the idea of allowing individual libraries and their dependencies being released between full Boost versions when deemed necessary.
I believe you misunderstood. Individual library packaging is not in scope of this proposal. I may be considered in a future. In a short term users can pull released individual library manually from svn. In both cases they will do it on their own risk. The only reliable way to use boost to stick with boost umbrella releases. Gennadiy
Gennadiy Rozental wrote:
"Edward Diener" <eldiener@tropicsoft.com> wrote in message news:f496rp$she$1@sea.gmane.org...
Gennadiy Rozental wrote:
Hi,
Following David A. recommendation I put up a wiki page that presents initial version of my proposal for boost development environment, testing and release procedures.
http://svn.boost.org/trac/boost/wiki/IndependentLibraryVersioning
It's still missing one key piece - version selection algorithm. I just don't have time to complete it now and won't until weekend. But it should be complete enough to explain all the key points (and some more). I shamelessly stole couple statements from Beman's objective section. Well, we do have similar objectives ;)
Any comments are welcome. I like the idea of allowing individual libraries and their dependencies being released between full Boost versions when deemed necessary.
I believe you misunderstood. Individual library packaging is not in scope of this proposal. I may be considered in a future. In a short term users can pull released individual library manually from svn. In both cases they will do it on their own risk. The only reliable way to use boost to stick with boost umbrella releases.
I sure did misunderstand your proposal, then. Your document, with a title of Independent Library Versioning, and a subject matter generally discussing the same, would fool most anyone into believing that you were talking about releasing individual Boost libraries separate from a Boost general release.
"Edward Diener" <eldiener@tropicsoft.com> wrote in message news:f49vdf$lvj$1@sea.gmane.org...
I believe you misunderstood. Individual library packaging is not in scope of this proposal. I may be considered in a future. In a short term users can pull released individual library manually from svn. In both cases they will do it on their own risk. The only reliable way to use boost to stick with boost umbrella releases.
I sure did misunderstand your proposal, then. Your document, with a title of Independent Library Versioning, and a subject matter generally discussing the same, would fool most anyone into believing that you were talking about releasing individual Boost libraries separate from a Boost general release.
No. It's talking about versioning libraries independently. Primarily to support independent concurrent libraries development. Packaging these intermittent releases in another issue that can be considered at later stage. Genandiy
2007/6/7, Gennadiy Rozental <gennadiy.rozental@thomson.com>:
I put up a wiki page that presents initial version of my proposal for boost development environment, testing and release procedures.
http://svn.boost.org/trac/boost/wiki/IndependentLibraryVersioning
Is the svn structure verified to work? <theProposal> Optional extensions/Combined tree as a reflection: To support the need for a single command update of several (all) boost libraries it might be convenient to create combined boost tree (similar to our current tree) using svn externals based reflections. boost/ lib1 -> lib1/trunk/boost/lib1 lib1.hpp -> lib1/trunk/boost/lib1.hpp lib2 -> lib2/trunk/boost/lib2 lib2.hpp -> lib2/trunk/boost/lib2.hpp lib3 -> lib3/trunk/boost/lib3 lib3.hpp -> lib3/trunk/boost/lib3.hpp </theProposal> I agree that this is needed. But I don't think it works.
From http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.advanced.externals "[A]n externals definition can only point to directories, not files." I.e. The hpp-externals above should not work.
I think this means that to be able to place all hpp-files in the boost-directory, there should be a build step to put them there. I..e When building lib1, it should copy it's own headers to the Boost directory. I don't know if that is good either. What happens if hpp-files are renamed? Will the old files not remain in the Boost directory? Is this a drawback in all proposals so far? /$
Henrik Sundberg wrote:
2007/6/7, Gennadiy Rozental <gennadiy.rozental@thomson.com>:
I put up a wiki page that presents initial version of my proposal for boost development environment, testing and release procedures.
http://svn.boost.org/trac/boost/wiki/IndependentLibraryVersioning
Is the svn structure verified to work? [-] I agree that this is needed. But I don't think it works.
From http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.advanced.externals "[A]n externals definition can only point to directories, not files." I.e. The hpp-externals above should not work.
I think this means that to be able to place all hpp-files in the boost-directory, there should be a build step to put them there. I..e When building lib1, it should copy it's own headers to the Boost directory. I don't know if that is good either. What happens if hpp-files are renamed? Will the old files not remain in the Boost directory?
Is this a drawback in all proposals so far?
I don't think Beman's proposal relies on any particular svn features. Also svn externals break the read-only (http) vs. read-write (https) arrangement. Someone who is using read-only access will not be able to do updates if put in externals of the https variety, and vice versa. In general I think relying on any particular RCS feature is going to get us into trouble down the road. But that's just a hunch, so don't put too much weight on it. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo
Rene Rivera wrote:
Is this a drawback in all proposals so far?
I don't think Beman's proposal relies on any particular svn features.
The essence of Beman's proposal is: a) maintain a releasable (stable) version. b) do developement on branches - one branch per development project. c) Test against the releasable version. d) Once a development project has been successfully tested against the the releasable version i) merge the development project into the releaseable version. ii) retest the whole releaseable version iii) automatically regenerate a deliverable package This is totally orthogonal to proposals to making independent library versions, changing the source control system, etc. We should try to change one thing at a time. I would suggest do the following in sequence and embarking on one thing only after the previous one has been digested. a) move to SVN - which seems already underway. b) implement to Beman's test/release proposal c) consider changes required to support independent library versioning. Robert Ramey
2007/6/10, Robert Ramey <ramey@rrsd.com>:
We should try to change one thing at a time. I would suggest do the following in sequence and embarking on one thing only after the previous one has been digested.
a) move to SVN - which seems already underway.
Yes, but a well defined structure is needed. We need to be able to check out the source in the defined structure, or to have tools adjusting the structure.
b) implement to Beman's test/release proposal c) consider changes required to support independent library versioning.
/$
Henrik Sundberg wrote:
2007/6/10, Robert Ramey <ramey@rrsd.com>:
We should try to change one thing at a time. I would suggest do the following in sequence and embarking on one thing only after the previous one has been digested.
a) move to SVN - which seems already underway.
Yes, but a well defined structure is needed. We need to be able to check out the source in the defined structure, or to have tools adjusting the structure.
We have a well defined structure. It may not be optimal - but that's another issue. As far as I'm concerned, SVN vs CVS totally separate from Beman's proposal. Given all the opportunities/issues raised by a move to SVN, maybe the change to SVN should be postponed until Beman's proposal is impemented. One thing is for sure - one thing at a time - and the most important thing first. Robert Ramey
b) implement to Beman's test/release proposal c) consider changes required to support independent library versioning.
/$ _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
We have a well defined structure. It may not be optimal - but that's another issue. As far as I'm concerned, SVN vs CVS totally separate from Beman's proposal. Given all the opportunities/issues raised by a move to SVN, maybe the change to SVN should be postponed until Beman's proposal is impemented.
No ! Since, as you say, the SVN <-> CVS change is orthogonal to the rest, and since changing structure is *much* more easy in SVN, there is no reason not to start with the SVN move. Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...
Stefan Seefeld wrote:
Robert Ramey wrote:
We have a well defined structure. It may not be optimal - but that's another issue. As far as I'm concerned, SVN vs CVS totally separate from Beman's proposal. Given all the opportunities/issues raised by a move to SVN, maybe the change to SVN should be postponed until Beman's proposal is impemented.
No ! Since, as you say, the SVN <-> CVS change is orthogonal to the rest, and since changing structure is *much* more easy in SVN, there is no reason not to start with the SVN move.
regardless of whether its easier or not with SVN or CVS or whatever, I'm arguing that the structure shouldn't be changed until the testing is fixed. Robert Ramey
Regards, Stefan
Robert Ramey <ramey <at> rrsd.com> writes: [...]
The essence of Beman's proposal is:
a) maintain a releasable (stable) version. b) do developement on branches - one branch per development project.
OK, but how? I'm not a subversion expert, but as far as I can tell the following is a reasonable description of what we're facing. If you branch the whole of Boost, developers will have to perform merges each time they want to update their sandbox. These may be inexpensive in terms of repository disk space, but are a much greater PITA than sandbox updates. Otherwise you have to keep into account the fact that Subversion branches influence your directory structure. Branching only the library specific parts with the current structure leads to one of the following alternatives: A - Branch from top level with empty intermediate levels: boost +branches +SERIALIZATION_BRANCH +boost | +serialization +libs +serialization B - Branch as low down as possible: boost +boost | +serialization | +branches | +SERIALIZATION_BRANCH +libs +serialization +branches +SERIALIZATION_BRANCH None of these allow checking out with a single command, unless you resort to heavy use of svn:externals. In this situation the alternative structure that several people propose seems more in line with most of the proposed process changes.
c) Test against the releasable version. d) Once a development project has been successfully tested against the the releasable version i) merge the development project into the releaseable version. ii) retest the whole releaseable version iii) automatically regenerate a deliverable package
Again, how? The notion of a single library test queue requires a form of tool support we currently don't have and even if we had it, I'm not convinced it would be easy to make this scheme work given the current type of discretionary, voluntary testing we have. On the other hand unless additional resources are found there's little else that can be done, apart from a little rationalization and, certainly, fixing the current tools.
This is totally orthogonal to proposals to making independent library versions, changing the source control system, etc.
I believe I've shown that this is not entirely true, at least as far as version control is concerned. I agree that single library versioning, however desirable, is a different matter.
We should try to change one thing at a time. I would suggest do the following in sequence and embarking on one thing only after the previous one has been digested.
a) move to SVN - which seems already underway. b) implement to Beman's test/release proposal c) consider changes required to support independent library versioning.
I still think that what I suggest here: http://svn.boost.org/trac/boost/wiki/AlternateProcessProposal is close to the minimum change we must apply if we want things to take a different turn in 1.35 . Cheers, Nicola Musatti
Nicola Musatti wrote:
Robert Ramey <ramey <at> rrsd.com> writes: [...]
The essence of Beman's proposal is:
a) maintain a releasable (stable) version. b) do developement on branches - one branch per development project.
OK, but how? I'm not a subversion expert, but as far as I can tell the following is a reasonable description of what we're facing.
If you branch the whole of Boost, developers will have to perform merges each time they want to update their sandbox. These may be inexpensive in terms of repository disk space, but are a much greater PITA than sandbox updates.
Otherwise you have to keep into account the fact that Subversion branches influence your directory structure. Branching only the library specific parts with the current structure leads to one of the following alternatives:
A - Branch from top level with empty intermediate levels:
boost +branches +SERIALIZATION_BRANCH +boost | +serialization +libs +serialization
B - Branch as low down as possible:
boost +boost
+serialization +branches +SERIALIZATION_BRANCH +libs +serialization +branches +SERIALIZATION_BRANCH
None of these allow checking out with a single command, unless you resort to heavy use of svn:externals. In this situation the alternative structure that several people propose seems more in line with most of the proposed process changes.
Personally, I have no problem with branching the directories and files related to my project - even if they can't be done with a single command. I don't think this would require any change in directory structure with either CVS or SVN.
c) Test against the releasable version. d) Once a development project has been successfully tested against the the releasable version i) merge the development project into the releaseable version. ii) retest the whole releaseable version iii) automatically regenerate a deliverable package
Again, how? The notion of a single library test queue requires a form of tool support we currently don't have
true.
and even if we had it, I'm not convinced it would be easy to make this scheme work given the current type of discretionary, voluntary testing we have.
That would depend upon how it was implemented. On the
other hand unless additional resources are found there's little else that can be done, apart from a little rationalization and, certainly, fixing the current tools.
The resources dedicated to testing would be much less than now so more effective testing could be done. That is this system would test only that which would benefit from testing - much, much less that the current approach.
This is totally orthogonal to proposals to making independent library versions, changing the source control system, etc.
I believe I've shown that this is not entirely true, at least as far as version control is concerned.
Well, we have to disagree here.
I agree that single library versioning, however desirable, is a different matter.
Agree.
We should try to change one thing at a time. I would suggest do the following in sequence and embarking on one thing only after the previous one has been digested.
a) move to SVN - which seems already underway. b) implement to Beman's test/release proposal c) consider changes required to support independent library versioning.
I still think that what I suggest here: http://svn.boost.org/trac/boost/wiki/AlternateProcessProposal is close to the minimum change we must apply if we want things to take a different turn in 1.35 .
I skimmed your document and: a) I don't believe that beman's proposal requires changing the directory tree structure. b) I don't believe that bemans proposal requires a change to SVN. But my main object is that it is "top down" in that is imposes more rules and structure on the testing/release procedure. I think that our current problems stem from the same approach. I think that the system has to be designed to permit more flexibility - not less which I think is what Beman's proposal does.
Cheers, Nicola Musatti
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
Nicola Musatti wrote: [...]
On the other hand unless additional resources are found there's little else that can be done, apart from a little rationalization and, certainly, fixing the current tools.
The resources dedicated to testing would be much less than now so more effective testing could be done. That is this system would test only that which would benefit from testing - much, much less that the current approach.
In terms of total usage I agree, but for this to work a much higher degree of coordination is required. I don't think it may be achieved without a form of tight centralized control. [...]
I still think that what I suggest here: http://svn.boost.org/trac/boost/wiki/AlternateProcessProposal is close to the minimum change we must apply if we want things to take a different turn in 1.35 .
I skimmed your document and:
a) I don't believe that beman's proposal requires changing the directory tree structure. b) I don't believe that bemans proposal requires a change to SVN.
But my main object is that it is "top down" in that is imposes more rules and structure on the testing/release procedure. I think that our current problems stem from the same approach. I think that the system has to be designed to permit more flexibility - not less which I think is what Beman's proposal does.
I see a contradiction here: you ask for flexibility, yet you expect a much tighter coordination of activities. All the proposals I've seen either wish for dependency management tools that aren't there or expect dependency problems to somehow sort themselves out. Suppose you've made some changes, asked and obtained that tests be run on your code, they passed and you checked in your code. Then the same goes for, say, MPL, but their change breaks your code. How's that different from what we have now? How does a similar situation affect the test request queue? Do you stop everything until you and the MPL authors sort things out? In my opinion flexibility stems from the fact that with a much shorter release cycle skipping one is not going to be such a big deal, so developers will be faced with reasonable alternatives. For that to happen each cycle must be managed very rigorously. Cheers, Nicola Musatti
Nicola Musatti wrote:
The resources dedicated to testing would be much less than now so more effective testing could be done. That is this system would test only that which would benefit from testing - much, much less that the current approach.
In terms of total usage I agree, but for this to work a much higher degree of coordination is required. I don't think it may be achieved without a form of tight centralized control.
Again, total disgreement. I run boost serialization tests on my home machine with bjam. I just move to the serialization/test directory and invoke a batch file which invokes bjam. There is no reason that anyone one else couldn't do that on their own machine. The only "coordination" required is a means to request the test and return the results. There is no reason this process has to be "coordinated" with anyone else - except to agree that that the test is to be run after moving to the proper development branch. Beman's proposal doesn't detail the mechanism for requesting a test, and posting results - but I don't see that that is a big issue. The essential fact is that this testing can run totally independently of everyone else - in fact that is exactly what happens when I run a test on my own machine. I don't have to "coordinate" this with anyone and so far no one has complained.
I see a contradiction here: you ask for flexibility, yet you expect a much tighter coordination of activities.
I expect LESS coordination of activities. The problems we have now are due to the fact that each and every library has to be in exactly the same state - ready for next release - at the same moment. Beman's proposal only requires that each library be ready one at a time.
All the proposals I've seen either wish for dependency management tools that aren't there or expect dependency problems to somehow sort themselves out.
Beman's proposal make no mention of dependency management. Implicit in the proposal are the following dependency "rules" a) each library is consistent with the state of those in the current release. b) library developer's should not break a current interface. Now b) will create a problem for some developer's. Some just won't care that their quest for perfection will leave other developer's out on a limb. No one can change this. No one can enforce a rule that interfaces can't change. No system of "managing dependcies" can prevent this. Developer's that do this will find that users and other developers won't want to depend on their library and the library will fall into disuse. The best we can do is run the final test after each library is integrated into the release and reject it if it breaks its interfaces.
Suppose you've made some changes, asked and obtained that tests be run on your code, they passed and you checked in your code. Then the same goes for, say, MPL, but their change breaks your code.
The MPL developer shouldn't do that except for very unsual cases. Generally such changes will be detected when the final test is run and the requested integration would be denied as in includes interface breaking changes.
How's that different from what we have now?
Now there are no disencentives to interface breaking changes and they cause all sorts of havoc.
How does a similar situation affect the test request queue?
see above.
you stop everything until you and the MPL authors sort things out?
proposed integration of MPL update is denied due to interface breaking changes. If it is determined that the interface has to be broken, then other developer which rely upon will have to choose to change their code or elminate their dependency on the library.
In my opinion flexibility stems from the fact that with a much shorter release cycle skipping one is not going to be such a big deal, so developers will be faced with reasonable alternatives. For that to happen each cycle must be managed very rigorously.
That's whats causing the current difficulty. Working harder at it will just make it worse. Robert Ramey
Robert Ramey <ramey <at> rrsd.com> writes:
Nicola Musatti wrote:
[...]
In terms of total usage I agree, but for this to work a much higher degree of coordination is required. I don't think it may be achieved without a form of tight centralized control.
Again, total disgreement. I run boost serialization tests on my home machine with bjam. I just move to the serialization/test directory and invoke a batch file which invokes bjam. There is no reason that anyone one else couldn't do that on their own machine.
This is the part we agree on the most :-)
The only "coordination" required is a means to request the test and return the results. There is no reason this process has to be "coordinated" with anyone else - except to agree that that the test is to be run after moving to the proper development branch. Beman's proposal doesn't detail the mechanism for requesting a test, and posting results - but I don't see that that is a big issue.
The difference is this part is not taking place on your machine anymore, but on the regression test machines, otherwise how can you test your changes on primary platforms you don't have access to?. Access to these machines does require coordination. In my opinion Beman's and other proposals seem to assume that there won't be contention of resources here. This is not an assumption I'd draw from seeing how things are going now.
The essential fact is that this testing can run totally independently of everyone else - in fact that is exactly what happens when I run a test on my own machine. I don't have to "coordinate" this with anyone and so far no one has complained.
In the light of what I wrote above I don't see how this may be the case.
I see a contradiction here: you ask for flexibility, yet you expect a much tighter coordination of activities.
I expect LESS coordination of activities. The problems we have now are due to the fact that each and every library has to be in exactly the same state - ready for next release - at the same moment. Beman's proposal only requires that each library be ready one at a time.
I attribute this situation to the fact that we're performing a sort of continuous integration in two different environments at the same time (HEAD and RC branch) most of the time, without a mechanism or procedure to ensure that breakage is swiftly dealt with. This in turn is in part induced by the lack of private "repositories" for development work that isn't yet ready for for integration. In a way we would probably improve things by just stopping all regression tests on HEAD.
All the proposals I've seen either wish for dependency management tools that aren't there or expect dependency problems to somehow sort themselves out.
Beman's proposal make no mention of dependency management. Implicit in the proposal are the following dependency "rules"
a) each library is consistent with the state of those in the current release. b) library developer's should not break a current interface.
Now b) will create a problem for some developer's. Some just won't care that their quest for perfection will leave other developer's out on a limb. No one can change this. No one can enforce a rule that interfaces can't change. No system of "managing dependencies" can prevent this. Developer's that do this will find that users and other developers won't want to depend on their library and the library will fall into disuse. The best we can do is run the final test after each library is integrated into the release and reject it if it breaks its interfaces.
You seem to assume that libraries cannot improve. The way out of this would be to share and discuss one's intentions with one's users and then coordinate the introduction of such changing with the authors of libraries that depend on the changing interfaces. I don't think users would appreciate the introduction of duplicated functionality in family of libraries that, rightly or wrongly, is perceived as a mostly coherent whole. [...]
In my opinion flexibility stems from the fact that with a much shorter release cycle skipping one is not going to be such a big deal, so developers will be faced with reasonable alternatives. For that to happen each cycle must be managed very rigorously.
That's whats causing the current difficulty. Working harder at it will just make it worse.
This is where we disagree. I keep thinking that if self discipline were enough, there wouldn't be problems right now. Cheers, Nicola Musatti
Nicola Musatti wrote:
The only "coordination" required is a means to request the test and return the results. There is no reason this process has to be "coordinated" with anyone else - except to agree that that the test is to be run after moving to the proper development branch. Beman's proposal doesn't detail the mechanism for requesting a test, and posting results - but I don't see that that is a big issue.
The difference is this part is not taking place on your machine anymore, but on the regression test machines, otherwise how can you test your changes on primary platforms you don't have access to?. Access to these machines does require coordination. In my opinion Beman's and other proposals seem to assume that there won't be contention of resources here. This is not an assumption I'd draw from seeing how things are going now.
The essential fact is that this testing can run totally independently of everyone else - in fact that is exactly what happens when I run a test on my own machine. I don't have to "coordinate" this with anyone and so far no one has complained.
In the light of what I wrote above I don't see how this may be the case.
Here is an example of a system that would implement the testing required by Beman's proposal. I' m not proposing this system but I am just demonstrating that it is possible. a) a developer wants to test a particular combination of library, development branch, compiler, operating system, standard library, variant, etc and posts such request to a central list of such requests. b) someone is willing / interested in testing some some combination of library, development branch, compiler, operating system, standard library, variant, etc. (Of course he might use a "*" wild card for one of the test parameters. When he posts his willingness to test he is returned a list of tests to run. c) the tests are run on the tester's machine and the results uploaded to the central test repository and/or emailed to the requestor. So, the decisions about what to test and when to test it are handled "coordinated" without requiring developer's and testers to work in lockstep. Each library can be developed independently at its own pace.
I attribute this situation to the fact that we're performing a sort of continuous integration in two different environments at the same time (HEAD and RC branch) most of the time, without a mechanism or procedure to ensure that breakage is swiftly dealt with.
I think we mostly agree on the source of the problem. Beman's approach trys to keep breakage out of the system rather than making an ever more elaborate system to deal with it.
This in turn is in part induced by the lack of private "repositories" for development work that isn't yet ready for for integration. In a way we would probably improve things by just stopping all regression tests on HEAD.
Agreed - this testing is not helpful to developer's nor to users.
You seem to assume that libraries cannot improve.
Nope. I have concluded that breaking interfaces is a bad thing for users. It is a breach of trust between the library developer and library user. I presumes that users have nothing else to do but constantly tweak their own code to keep up with the libraries changes. This defeats the whole purpose of using a library in the first place. Library developers who do this will lose "customers". We have already seen examples of this. Having said that, boost library authors have been pretty good about not breaking interfaces so I don't think this will be a huge problem in practice. Libraries can improve implementations and can ADD interfaces with no problem. This works well with the system.
The way out of this would be to share and discuss one's intentions with one's users and then coordinate the introduction of such changing with the authors of libraries that depend on the changing interfaces.
Nope. User code is already written and running. Suppose there are 1000 users of the serialization library. Suppose I were to make a change for 1.34 that would require all those users to investigate their code and tweak it because I made a change? Do you really think that would be OK?
I don't think users would appreciate the introduction of duplicated functionality in family of libraries that, rightly or wrongly, is perceived as a mostly coherent whole.
Now that is the central question. "mostly coherent whole" highlights the issue. I would guess that when boost started out, the "model" that was envisioned was STL which is a very coherent whole. I think experience has shown that such a vision is a chimera and no amount of work will make such a vision reality. The next best thing (actually a better thing) - is a large set of decoupled libraries which have stable interfaces. Note the by "decoupled" I don't mean that a library should not depend upon another library but rather it should depend only upon its document interface. The best thing that boost - and only boost - can do is to use its review process to promote/enforce. a) conceptional integrity - libaries which do one thing at a time without mixing up bunch of orthogonal functionality. b) well defined and well documented interfaces. c) quality implementation d) quality documentation library "unit testing" against the released library enhances these goals.
[...]
In my opinion flexibility stems from the fact that with a much shorter release cycle skipping one is not going to be such a big deal, so developers will be faced with reasonable alternatives. For that to happen each cycle must be managed very rigorously.
That's whats causing the current difficulty. Working harder at it will just make it worse.
This is where we disagree. I keep thinking that if self discipline were enough, there wouldn't be problems right now.
I agree - its not about self discipline. Its not that people aren't working hard enough. As I said, working harder won't help. Its that we're doing the wrong thing. Robert Ramey
Robert Ramey wrote:
Nicola Musatti wrote: [...] Here is an example of a system that would implement the testing required by Beman's proposal. I' m not proposing this system but I am just demonstrating that it is possible.
[Reasonable scheme snipped] I see. A developer might have to wait a while for tests to happen, but there should be a gain in terms of much reduced confusion with respect to how things work now. And once tests pass s/he is guaranteed that s/he is done with that specific change.
So, the decisions about what to test and when to test it are handled "coordinated" without requiring developer's and testers to work in lockstep. Each library can be developed independently at its own pace.
Provided the queue doesn't grow to much. But then that would only mean that testing resources are insufficient, and there's very little that can be done about that apart from adding some other. [...]
I have concluded that breaking interfaces is a bad thing for users. It is a breach of trust between the library developer and library user. I presumes that users have nothing else to do but constantly tweak their own code to keep up with the libraries changes. This defeats the whole purpose of using a library in the first place. Library developers who do this will lose "customers". We have already seen examples of this. Having said that, boost library authors have been pretty good about not breaking interfaces so I don't think this will be a huge problem in practice.
I understand your feelings, although one could say that designing interfaces in view of standardization is one of Boost's goals. The Boost libraries are experimental, in a way. [...]
User code is already written and running. Suppose there are 1000 users of the serialization library. Suppose I were to make a change for 1.34 that would require all those users to investigate their code and tweak it because I made a change? Do you really think that would be OK?
Not unless there was a lot to be gained by the change. And even then what is a lot for someone is likely to be totally irrelevant for someone else.
I don't think users would appreciate the introduction of duplicated functionality in family of libraries that, rightly or wrongly, is perceived as a mostly coherent whole.
Now that is the central question. "mostly coherent whole" highlights the issue. I would guess that when boost started out, the "model" that was envisioned was STL which is a very coherent whole. I think experience has shown that such a vision is a chimera and no amount of work will make such a vision reality.
The next best thing (actually a better thing) - is a large set of decoupled libraries which have stable interfaces. Note the by "decoupled" I don't mean that a library should not depend upon another library but rather it should depend only upon its document interface.
Oh, absolutely! Common functionality across libraries should always emerge in the form of, possibly small, additional libraries. There shouldn't be a common 'detail' directory.
The best thing that boost - and only boost - can do is to use its review process to promote/enforce. a) conceptional integrity - libaries which do one thing at a time without mixing up bunch of orthogonal functionality. b) well defined and well documented interfaces. c) quality implementation d) quality documentation
library "unit testing" against the released library enhances these goals.
I agree.
[...]
In my opinion flexibility stems from the fact that with a much shorter release cycle skipping one is not going to be such a big deal, so developers will be faced with reasonable alternatives. For that to happen each cycle must be managed very rigorously. That's whats causing the current difficulty. Working harder at it will just make it worse. This is where we disagree. I keep thinking that if self discipline were enough, there wouldn't be problems right now.
I agree - its not about self discipline. Its not that people aren't working hard enough. As I said, working harder won't help. Its that we're doing the wrong thing.
I still think I have to see it working to be completely convinced, but I'd welcome being proved wrong :-) My only remaining question - not specifically to you, Robert - is how is this whole thing started? Cheers, Nicola Musatti
2007/6/10, Rene Rivera <grafikrobot@gmail.com>:
Henrik Sundberg wrote:
I think this means that to be able to place all hpp-files in the boost-directory, there should be a build step to put them there. I..e When building lib1, it should copy it's own headers to the Boost directory. I don't know if that is good either. What happens if hpp-files are renamed? Will the old files not remain in the Boost directory?
Is this a drawback in all proposals so far?
I don't think Beman's proposal relies on any particular svn features.
Do you mean this, or is it defined at some other place? http://mysite.verizon.net/beman/development_environment.html#Repository
Also svn externals break the read-only (http) vs. read-write (https) arrangement. Someone who is using read-only access will not be able to do updates if put in externals of the https variety, and vice versa.
Troy seems to have a very functional structure, but I don't think he places files from different libraries in the same directory. /$
on Sun Jun 10 2007, Rene Rivera <grafikrobot-AT-gmail.com> wrote:
I don't think Beman's proposal relies on any particular svn features. Also svn externals break the read-only (http) vs. read-write (https) arrangement. Someone who is using read-only access will not be able to do updates if put in externals of the https variety, and vice versa.
We can provide anonymous read-only access via https. That's what my repository does. Easy as pie. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
on Sun Jun 10 2007, "Henrik Sundberg" <storangen-AT-gmail.com> wrote:
2007/6/7, Gennadiy Rozental <gennadiy.rozental@thomson.com>:
I put up a wiki page that presents initial version of my proposal for boost development environment, testing and release procedures.
http://svn.boost.org/trac/boost/wiki/IndependentLibraryVersioning
Is the svn structure verified to work?
<theProposal> Optional extensions/Combined tree as a reflection: To support the need for a single command update of several (all) boost libraries it might be convenient to create combined boost tree (similar to our current tree) using svn externals based reflections. boost/ lib1 -> lib1/trunk/boost/lib1 lib1.hpp -> lib1/trunk/boost/lib1.hpp lib2 -> lib2/trunk/boost/lib2 lib2.hpp -> lib2/trunk/boost/lib2.hpp lib3 -> lib3/trunk/boost/lib3 lib3.hpp -> lib3/trunk/boost/lib3.hpp </theProposal>
I agree that this is needed. But I don't think it works.
From http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.advanced.externals "[A]n externals definition can only point to directories, not files." I.e. The hpp-externals above should not work.
I think this means that to be able to place all hpp-files in the boost-directory, there should be a build step to put them there. I..e When building lib1, it should copy it's own headers to the Boost directory. I don't know if that is good either. What happens if hpp-files are renamed? Will the old files not remain in the Boost directory?
Is this a drawback in all proposals so far? /$
The way to handle this, IMO, is to: a. move all headers into a library-specific subdirectories b. for backward-compatibility purposes, keep minimalist forwarding headers in the main distribution. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
participants (8)
-
David Abrahams -
Edward Diener -
Gennadiy Rozental -
Henrik Sundberg -
Nicola Musatti -
Rene Rivera -
Robert Ramey -
Stefan Seefeld