From another thread: Steven Watanabe wrote:
AMDG
troy d. straszheim wrote:
I've been wondering a bit about this in-syncness recently.
- Does anybody know offhand what the last release was that was branched directly from trunk?
- Is there a way to determine, with SVN, how many lines of diff there are between trunk and release? I've tried without success.... the diff gets to ~80k lines and then the network times out. I wonder if I'm doing something wrong. Of course I can get some numbers with git but I'm not quite sure I trust them yet.
% git diff --shortstat svn/release svn/master 3505 files changed, 209061 insertions(+), 58428 deletions(-)
This is probably correct as far as it goes. However, it's hard to tell how much it really means. About half the modified files seem to be generated html, judging by the output of
svn diff https://svn.boost.org/svn/boost/branches/release https://svn.boost.org/svn/boost/trunk --summarize
Hm, that svn command isn't very helpful and runs for ~5 minutes. I'm seeing about 160k lines total changed in actual code (.cpp, .ipp, .hpp). Full report in case anybody is interested: http://sodium.resophonic.com/trunk_vs_release.txt Poking around it looks like the current release branch was created from the fixed 1.34.1 tag in oct 2007 (@ r40341). The 1.34.1 release was in turn branched from... looks like that goes back into the days of CVS, I'll consider it lost. Git maintains that the last common commit between the current trunk and release branches is > 3 years ago: http://gitorious.org/boost/svn/commit/4914d4934fbc7368e2d43e057f6fffdac0f464... So if the trunk is really just a staging area for merges to release, does the notion of having the release branch and trunk 'in sync' at this point make any sense? Why have just one such staging area? -t
Daniel James wrote:
2009/5/21 troy d. straszheim <troy@resophonic.com>:
Why have just one such staging area?
So we don't have to run regression tests on lots of branches.
Okay, but with the current trunk, you're regression testing a lot of cruft that has been sitting there ignored for years. How about regression-testing a codebase that has a fighting chance of actually becoming a release? -t
2009/5/21 troy d. straszheim <troy@resophonic.com>:
Okay, but with the current trunk, you're regression testing a lot of cruft that has been sitting there ignored for years.
Whatever branching scheme you use, unmaintained code will still be unmaintained.
How about regression-testing a codebase that has a fighting chance of actually becoming a release?
Boost is a volunteer organisation. In order for something to happen that effects everyone, someone has to make a proposal, convince people (I'm not sure exactly who) to go along with it, deal with whatever issues there are and put it into action. And in this case, nobody has. Daniel
Daniel James wrote:
2009/5/21 troy d. straszheim <troy@resophonic.com>:
Okay, but with the current trunk, you're regression testing a lot of cruft that has been sitting there ignored for years.
Whatever branching scheme you use, unmaintained code will still be unmaintained.
That's true of course. And of course this has little to do with the practice of repeatedly testing and 'merging around' unmaintained, unreleased code.
How about regression-testing a codebase that has a fighting chance of actually becoming a release?
Boost is a volunteer organisation.
You're stating the obvious; sorry if I'm irritating you. (BTW I didn't mean "you" above when I said "you're regression testing"). I'm just being careful, trying to establish how much of this is by design, and how much has just 'evolved' on its own...
In order for something to happen that effects everyone, someone has to make a proposal, convince people (I'm not sure exactly who) to go along with it, deal with whatever issues there are and put it into action. And in this case, nobody has.
... and now I have my answer. Thanks. -t
2009/5/21 troy d. straszheim <troy@resophonic.com>:
That's true of course. And of course this has little to do with the practice of repeatedly testing and 'merging around' unmaintained, unreleased code.
Generally, it isn't that much of a problem, since most developers just deal with a couple of directories. It isn't great but for the most part we can just ignore most of boost. I suppose if you're working on a new build system you can't.
How about regression-testing a codebase that has a fighting chance of actually becoming a release?
Boost is a volunteer organisation.
You're stating the obvious; sorry if I'm irritating you.
Sorry, my mail was too negative. We get a lot of 'somebody should do something' emails. You are doing something with your cmake work, so I wasn't being fair at all. If that results in a better testing system and better modularization that could improve the situation massively - having a single tree for such disparate libraries that are developed at different rates doesn't help.
(BTW I didn't mean "you" above when I said "you're regression testing"). I'm just being careful, trying to establish how much of this is by design, and how much has just 'evolved' on its own...
It hasn't entirely evolved on its own. A few years back we the release and trunk branches were a complete mess (the current situation looks very clean in comparison). We cleaned up a lot of the conflicts and established the rule that changes had to be made to trunk and go through regression testing before being merged to release. Which has made things a lot better. It isn't ideal but it works a lot better than what we used to have which might be why we generally feel content. The problem is that we can't branch release from trunk - as we don't know what new stuff is release worthy. And we don't want to overwrite trunk from release as that might lose something and it would interfere with more long term development. Working on a branch is a nice idea in theory, in practice it turns out to be a pain since boost is so large, especially with subversion. And we really do need to get code into the regression testing system as soon as possible. With git it is a little better as you can create a micro branch and rebase it, but git isn't really in a state where most boost developers could use it. Modularization would help a lot. If you look at your summary of differences, many of them are in Boost.Spirit and Boost.Math which are well maintained and are perhaps long term changes that aren't ready for release yet. But there are a lot of small changes in largely unmaintained libraries - I think they just need people to go through them and clean them up. I think we are able to automatically identify which libraries most headers belong to, so a tool which summarized the unchanged headers might help. Anything which is left unmerged and untouched for some time would be a cause for concern. Daniel
Going back to a very early post, Daniel James wrote:
2009/5/21 troy d. straszheim <troy@resophonic.com>:
That's true of course. And of course this has little to do with the practice of repeatedly testing and 'merging around' unmaintained, unreleased code.
Generally, it isn't that much of a problem, since most developers just deal with a couple of directories. It isn't great but for the most part we can just ignore most of boost. I suppose if you're working on a new build system you can't.
I think there are a variety of situations in which one can't afford to ignore most of boost, or deal with only one 'component' at a time. Boost 'power users' (I term I've started using and am sure will grow to hate, apologies in advance) may want to maintain patchsets against the entire trunk, for instance. A GSOC student who is doing a port of fusion to c++0x might temporarily need any number of workarounds sprinkled throughout boost.
Sorry, my mail was too negative.
I'm finding your follow-up very productive (thanks), no worries.
[snip] If that results in a better testing system and better modularization, that could improve the situation massively - having a single tree for such disparate libraries that are developed at different rates doesn't help.
There's an ongoing assumption that modularization will help. Some of it I am fuzzy on. It helps to have headers and src under the same directory, I think there is wide agreement on that, but why? - it enables one to more easily trim down boost to prepare subdistributions. Uncontroversial. - it enables you to put boost components on the ends of svn:externals and version them separately: very controversial. And the one I'm interested in discussing now: - it enables one to more easily 'switchmerge' a single library. This is true. But is the need for this just a consequence of the fact that merging in svn is onerous/broken, no?
I'm just being careful, trying to establish how much of this is by design, and how much has just 'evolved' on its own...
It hasn't entirely evolved on its own. A few years back we the release and trunk branches were a complete mess (the current situation looks very clean in comparison). We cleaned up a lot of the conflicts and established the rule that changes had to be made to trunk and go through regression testing before being merged to release. Which has made things a lot better. It isn't ideal but it works a lot better than what we used to have which might be why we generally feel content.
Yes clearly an improvement.
The problem is that we can't branch release from trunk - as we don't know what new stuff is release worthy. And we don't want to overwrite trunk from release as that might lose something and it would interfere with more long term development.
It seems clear that the problem is lack of granularity. The trunk is a rather full, messy waiting room for the release branch. As the runup to a release comes, the release manager has little say in what uses testing resources.
Working on a branch is a nice idea in theory, in practice it turns out to be a pain since boost is so large, especially with subversion.
Is the prioritization backwards here? The linux kernel is twice as big. They can merge in seconds, and they do, (on the order of 4x/day iirc).
And we really do need to get code into the regression testing system as soon as possible. With git it is a little better as you can create a micro branch and rebase it, but git isn't really in a state where most boost developers could use it. Modularization would help a lot.
Let me push back at you a bit here. It is easy to provide a git setup that behaves nearly identically to subversion, if one wants to do things in a centralized way, ie just checkout, update, local diff, commit. Rebasing: nice in that it allows one to forward a nice linear history to the upstream repository, but IIUC this was developed only after the linux kernel folks realized that looking at commit history was actually useful, and they started to have opinions on what looks good. In any event, the fact that code has been rebased is invisible to public repositories. As I currently see it, the primitive we lack is the release/testing manager's ability to efficiently *merge* the next chunk of code from downstream into the testing queue. Whether the code he's merging is the result of nice clean rebases or a spiderweb of merges is irrelevant, at least until people start to care about how merge history looks (which may be never). So, does modularization help a lot because it allows one to merge by copying directories around? It does appears to have other costs, for instance having to use scripts to construct include directories full of forwarding headers, potentially combined with walls of svn:externals, etc.
If you look at your summary of differences, many of them are in Boost.Spirit and Boost.Math which are well maintained and are perhaps long term changes that aren't ready for release yet. But there are a lot of small changes in largely unmaintained libraries - I think they just need people to go through them and clean them up.
I think we are able to automatically identify which libraries most headers belong to, so a tool which summarized the unchanged headers might help. Anything which is left unmerged and untouched for some time would be a cause for concern.
Right, there are ways to get in there and clean up the trunk, this is tractable. But it seems that if the system remains unchanged, the situation will naturally evolve again. Thanks again for your time and patience, -t
2009/5/25 troy d. straszheim <troy@resophonic.com>:
[snip] If that results in a better testing system
and better modularization, that could improve the situation massively - having a single tree for such disparate libraries that are developed at different rates doesn't help.
There's an ongoing assumption that modularization will help.
FWIW, I'm more interested in the testing system. [snip]
And the one I'm interested in discussing now:
- it enables one to more easily 'switchmerge' a single library. This is true. But is the need for this just a consequence of the fact that merging in svn is onerous/broken, no?
I don't think that's a big deal - if a library is orgranised well the current system only requires two individual directories to be switched (although I use svnmerge myself). [snip]
It seems clear that the problem is lack of granularity. The trunk is a rather full, messy waiting room for the release branch. As the runup to a release comes, the release manager has little say in what uses testing resources.
Sure, but with our current testing and maintenance situation, I can't see a realistic way around this. [snip]
Is the prioritization backwards here? The linux kernel is twice as big. They can merge in seconds, and they do, (on the order of 4x/day iirc).
There's also a lot more effort and resources going into kernel development.
[snipped a lot about git]
The major stumbling block is that git isn't good enough on windows. Mercurial might be a better candidate, since it has TortoiseHg. Although I haven't tried it myself. There's also the administrative burden of moving to a new version control system - we haven't even managed to upgrade from subversion 1.4 yet. And I think a lot of boost developers just aren't that interested in version control. Whatever we use will have to be usable with the minimum of effort. I wrote:
I think we are able to automatically identify which libraries most headers belong to, so a tool which summarized the unchanged headers might help. Anything which is left unmerged and untouched for some time would be a cause for concern.
I think I was being overly optimistic here. Without merge tracking it's hard to work out what's what.
Right, there are ways to get in there and clean up the trunk, this is tractable. But it seems that if the system remains unchanged, the situation will naturally evolve again.
But I don't think it's that bad. It's yet to cause me any problems with my libraries, although that might be because all of their dependencies are very stable. Daniel
This discussion demonstrates what's wrong with the current testing of the trunk. Here is the way it should be done. For each library that is changed, the library version on the trunk should be tested against the release branch. In practice this would work something like the following: On the tester's machine start out with a current image of the release branch. For each library A switch library A's directories to the trunk run tests Restore image to release This would a) attribute failures to their proper source b) prevent testing of each library (A) with provisional (trunk) code c) guarentee that when changes are merged into the release no new failures occur. d) reduce testing time since a small expermental change in one library in the trunk wouldn't provoke rebuild and test of all its dependents. On my own machine, I have a release branch and I've switched the serialization library directories to the trunk. I can verify that all the above mentioned benefits are in fact realized. Boost testing would benefit greatly from adopting the same strategy. Robert Ramey troy d. straszheim wrote:
Daniel James wrote:
2009/5/21 troy d. straszheim <troy@resophonic.com>:
Okay, but with the current trunk, you're regression testing a lot of cruft that has been sitting there ignored for years.
Whatever branching scheme you use, unmaintained code will still be unmaintained.
That's true of course. And of course this has little to do with the practice of repeatedly testing and 'merging around' unmaintained, unreleased code.
How about regression-testing a codebase that has a fighting chance of actually becoming a release?
Boost is a volunteer organisation.
You're stating the obvious; sorry if I'm irritating you. (BTW I didn't mean "you" above when I said "you're regression testing"). I'm just being careful, trying to establish how much of this is by design, and how much has just 'evolved' on its own...
In order for something to happen that effects everyone, someone has to make a proposal, convince people (I'm not sure exactly who) to go along with it, deal with whatever issues there are and put it into action. And in this case, nobody has.
... and now I have my answer. Thanks.
-t
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Thu, May 21, 2009 at 12:45 PM, Robert Ramey <ramey@rrsd.com> wrote:
This discussion demonstrates what's wrong with the current testing of the trunk.
Here is the way it should be done.
For each library that is changed, the library version on the trunk should be tested against the release branch. In practice this would work something like the following:
On the tester's machine start out with a current image of the release branch. For each library A switch library A's directories to the trunk run tests Restore image to release
Yep, that would often be optimal. Without spending a lot of time thinking it through, that sounds roughly like the test-on-demand scenario we've often talked about, except with test-on-demand it might be any branch, not just trunk, that gets switched to. One issue, however, is when a library that other libraries depend on makes a breaking change. In that case a set of libraries have to be switched together for testing, and then merged into release at the same time. --Beman
Beman Dawes wrote:
On Thu, May 21, 2009 at 12:45 PM, Robert Ramey <ramey@rrsd.com> wrote:
This discussion demonstrates what's wrong with the current testing of the trunk.
Here is the way it should be done.
For each library that is changed, the library version on the trunk should be tested against the release branch. In practice this would work something like the following:
On the tester's machine start out with a current image of the release branch. For each library A switch library A's directories to the trunk run tests Restore image to release
Yep, that would often be optimal. Without spending a lot of time thinking it through, that sounds roughly like the test-on-demand scenario we've often talked about, except with test-on-demand it might be any branch, not just trunk, that gets switched to.
Suppose that it was a large loop: for each library switch directories from release to trunk run bjam from within that directory since bjam tracks dependencies if there is no change no tests are run if there are some changes only the dependent tests are run switch library directories back to release on to the next library Testers would run the whole loop occasionally. My guess is that the total amount of time required would be similar to running the current test setup or maybe it would be a lot less.
One issue, however, is when a library that other libraries depend on makes a breaking change.
This should be infrequent and would be a special case.
In that case a set of libraries have to be switched together for testing, and then merged into release at the same time.
I don't think so. Note that in the above regimen, such a breaking change (intentional or not) would appear as a whole slew of failures in dependent libraries as soon as the breaking change is merged into the release. At that point they dependent libraries would have to be fixed. But this doesn't affect the testing in any way. It just slogs on with more failures until they are fixed and the the fixed libraries are merged into the release. Robert Ramey
--Beman _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
This discussion demonstrates what's wrong with the current testing of the trunk.
Here is the way it should be done.
For each library that is changed, the library version on the trunk should be tested against the release branch.
I fully agree. This requires boost to become modular, with dependencies between libraries properly being tracked and cared for, such that such an arrangement becomes possible. (There are many more advantages to a modular boost, which I won't repeat here, as we have had such discussions multiple times in the past, never getting anywhere.) Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...
On Thu, May 21, 2009 at 12:08 PM, Stefan Seefeld <seefeld@sympatico.ca> wrote:
Robert Ramey wrote:
This discussion demonstrates what's wrong with the current testing of the trunk.
Here is the way it should be done.
For each library that is changed, the library version on the trunk should be tested against the release branch.
I fully agree. This requires boost to become modular, with dependencies between libraries properly being tracked and cared for, such that such an arrangement becomes possible. (There are many more advantages to a modular boost, which I won't repeat here, as we have had such discussions multiple times in the past, never getting anywhere.)
There was a lot of discussion of modularization at BoostCon. Several of us have volunteered our libraries as guinea pigs once the mechanisms are in place. --Beman
Beman Dawes wrote:
On Thu, May 21, 2009 at 12:08 PM, Stefan Seefeld <seefeld@sympatico.ca> wrote:
Robert Ramey wrote:
This discussion demonstrates what's wrong with the current testing of the trunk.
Here is the way it should be done.
For each library that is changed, the library version on the trunk should be tested against the release branch.
I fully agree. This requires boost to become modular, with dependencies between libraries properly being tracked and cared for, such that such an arrangement becomes possible. (There are many more advantages to a modular boost, which I won't repeat here, as we have had such discussions multiple times in the past, never getting anywhere.)
There was a lot of discussion of modularization at BoostCon. Several of us have volunteered our libraries as guinea pigs once the mechanisms are in place.
FWIW, I expect Boost.Build mechanisms to be implemented shortly. However, to be fully sensible, it will require change of layout of libraries in SVN -- otherwise we'll end up in a situation currently with CMake -- where there's 'modularize' command which changes directory layout to be different from that in SVN -- which is not a good idea in general. - Volodya
Vladimir Prus wrote:
FWIW, I expect Boost.Build mechanisms to be implemented shortly. However, to be fully sensible, it will require change of layout of libraries in SVN -- otherwise we'll end up in a situation currently with CMake -- where there's 'modularize' command which changes directory layout to be different from that in SVN -- which is not a good idea in general.
But we can easily agree on what that layout should look like, and switch both build systems simultaneously, right? Then we can do away with the 'modularize' business. Let's do it! Hm, wait: 1. those who develop against checkouts of boost (not installed versions) will need to define just what boost components they need. It could get messy, but it has to be done. 2. We'll have duplicated dependency information (see libs/*/module.cmake... presumably there will be some analogue on the boost.build side) 3. We've said nothing about defining version-specific dependencies (boost.spirit v.X depends on boost.proto >= v.Y) Just thinking out loud, -t
troy d. straszheim wrote:
Vladimir Prus wrote:
FWIW, I expect Boost.Build mechanisms to be implemented shortly. However, to be fully sensible, it will require change of layout of libraries in SVN -- otherwise we'll end up in a situation currently with CMake -- where there's 'modularize' command which changes directory layout to be different from that in SVN -- which is not a good idea in general.
But we can easily agree on what that layout should look like, and switch both build systems simultaneously, right? Then we can do away with the 'modularize' business. Let's do it!
Hm, wait:
1. those who develop against checkouts of boost (not installed versions) will need to define just what boost components they need. It could get messy, but it has to be done.
I am sure that folks who use either boost.build or cmake can be provided an easy way to get all components and their includes. However, we have 91 libraries at present. Adding 91 includes path to command line seems like extremely bad idea -- it will overflow windows command line limit easily, and gcc only recently got response files support, so probably does not have it in mingw and cygwin. Maybe we should review the goals of modularization? For me, its value is mostly that: 1. You can switch to a different version of a given library using single command. 2. It is possible to easily grab a subset of libraries. Maybe, we can still have single include directory while achieving the above goal if we make build process create symlinks to directories? That works on Linux, and I believe modern windows has some way to create links too. I am worried about other operating systems, though.
2. We'll have duplicated dependency information (see libs/*/module.cmake... presumably there will be some analogue on the boost.build side)
Can we create some shared file? I think both build system can read&parse a file.
3. We've said nothing about defining version-specific dependencies (boost.spirit v.X depends on boost.proto >= v.Y)
Do we even want to open this can of worms? - Volodya
Vladimir Prus wrote:
3. We've said nothing about defining version-specific dependencies (boost.spirit v.X depends on boost.proto >= v.Y)
Do we even want to open this can of worms?
While this sounds like a plausible use-case, I think the first step still remains to get a handle on dependencies, without versioning. So I wouldn't worry about that case just yet. :-) Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...
Vladimir Prus wrote:
troy d. straszheim wrote:
Vladimir Prus wrote:
FWIW, I expect Boost.Build mechanisms to be implemented shortly. However, to be fully sensible, it will require change of layout of libraries in SVN -- otherwise we'll end up in a situation currently with CMake -- where there's 'modularize' command which changes directory layout to be different from that in SVN -- which is not a good idea in general.
But we can easily agree on what that layout should look like, and switch both build systems simultaneously, right? Then we can do away with the 'modularize' business. Let's do it!
Hm, wait:
1. those who develop against checkouts of boost (not installed versions) will need to define just what boost components they need. It could get messy, but it has to be done.
I am sure that folks who use either boost.build or cmake can be provided an easy way to get all components and their includes. However, we have 91 libraries at present. Adding 91 includes path to command line seems like extremely bad idea -- it will overflow windows command line limit easily, and gcc only recently got response files support, so probably does not have it in mingw and cygwin.
I'm worried about the include paths too. Pondering modularization: It isn't clear to me that e.g. 'flyweight' and 'util' need to be entirely modularized. Maybe we could go coarser. But I'm not sure where I'm going with this.
Maybe we should review the goals of modularization? For me, its value is mostly that: 1. You can switch to a different version of a given library using single command. 2. It is possible to easily grab a subset of libraries.
I agree with these.... here's some specific cases: - be able to play with proposed/sandbox projects without downloading a zip file from the vault (crucial -- special case of 1.). - be able to assemble a boost piecemeal... say you need serialization and python.
Maybe, we can still have single include directory while achieving the above goal if we make build process create symlinks to directories? That works on Linux, and I believe modern windows has some way to create links too. I am worried about other operating systems, though.
I share your worry. Probably some scm mechanism should take care of it. I'm out of ideas for the moment.
2. We'll have duplicated dependency information (see libs/*/module.cmake... presumably there will be some analogue on the boost.build side)
Can we create some shared file? I think both build system can read&parse a file.
Sure.
3. We've said nothing about defining version-specific dependencies (boost.spirit v.X depends on boost.proto >= v.Y)
Do we even want to open this can of worms?
I'm not sure we do. Of course we are implicitly assembling these dependency graphs as we do releases. It may actually be possible to ferret out the details automatically by trawling through version control. A nice project for a saturday night... -t
troy d. straszheim wrote:
Vladimir Prus wrote:
troy d. straszheim wrote:
Vladimir Prus wrote:
FWIW, I expect Boost.Build mechanisms to be implemented shortly. However, to be fully sensible, it will require change of layout of libraries in SVN -- otherwise we'll end up in a situation currently with CMake -- where there's 'modularize' command which changes directory layout to be different from that in SVN -- which is not a good idea in general.
But we can easily agree on what that layout should look like, and switch both build systems simultaneously, right? Then we can do away with the 'modularize' business. Let's do it!
Hm, wait:
1. those who develop against checkouts of boost (not installed versions) will need to define just what boost components they need. It could get messy, but it has to be done.
I am sure that folks who use either boost.build or cmake can be provided an easy way to get all components and their includes. However, we have 91 libraries at present. Adding 91 includes path to command line seems like extremely bad idea -- it will overflow windows command line limit easily, and gcc only recently got response files support, so probably does not have it in mingw and cygwin.
I'm worried about the include paths too. Pondering modularization: It isn't clear to me that e.g. 'flyweight' and 'util' need to be entirely modularized. Maybe we could go coarser. But I'm not sure where I'm going with this.
Maybe we should review the goals of modularization? For me, its value is mostly that: 1. You can switch to a different version of a given library using single command. 2. It is possible to easily grab a subset of libraries.
I agree with these.... here's some specific cases:
- be able to play with proposed/sandbox projects without downloading a zip file from the vault (crucial -- special case of 1.).
- be able to assemble a boost piecemeal... say you need serialization and python.
That's subset of (2) above -- I think :-)
Maybe, we can still have single include directory while achieving the above goal if we make build process create symlinks to directories? That works on Linux, and I believe modern windows has some way to create links too. I am worried about other operating systems, though.
I share your worry. Probably some scm mechanism should take care of it. I'm out of ideas for the moment.
svn:externals in SVN allow you to have master copies of headers say in libs/thread/include and have a copy checked out into boost/thread while making commits in boost/thread work, and commit into the master copy. Disk requirements are a question, though.
3. We've said nothing about defining version-specific dependencies (boost.spirit v.X depends on boost.proto >= v.Y)
Do we even want to open this can of worms?
I'm not sure we do. Of course we are implicitly assembling these dependency graphs as we do releases. It may actually be possible to ferret out the details automatically by trawling through version control. A nice project for a saturday night...
There are different issues of: 1. Permitting to create Boost 1.Frankenstein release including random versions of all libraries, while somehow checking the requirements above. 2. Permitting to have different version of the same library in such combined package. Probably some of those can be good for like boost.serialization needing spirit 1.6 -- but I am not sure that use case is still very important, and whether we have any other similar. - Volodya
On Thu, May 21, 2009 at 2:40 PM, Vladimir Prus <vladimir@codesourcery.com>wrote:
troy d. straszheim wrote:
I share your worry. Probably some scm mechanism should take care of it. I'm out of ideas for the moment.
svn:externals in SVN allow you to have master copies of headers say in
libs/thread/include
and have a copy checked out into
boost/thread
while making commits in boost/thread work, and commit into the master copy. Disk requirements are a question, though.
I also really worried about this header issue. Volodya, thanks for the svn:externals suggestion. I'll try to learn more about svn:externals so I can form an opinion. --Beman
Beman Dawes wrote:
On Thu, May 21, 2009 at 2:40 PM, Vladimir Prus <vladimir@codesourcery.com>wrote:
troy d. straszheim wrote:
I share your worry. Probably some scm mechanism should take care of it. I'm out of ideas for the moment. svn:externals in SVN allow you to have master copies of headers say in
libs/thread/include
and have a copy checked out into
boost/thread
while making commits in boost/thread work, and commit into the master copy. Disk requirements are a question, though.
I also really worried about this header issue.
Volodya, thanks for the svn:externals suggestion. I'll try to learn more about svn:externals so I can form an opinion.
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional wisdom. It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project. You can end up needing to write tools to wrangle your externals. It won't be too hard to make a sample implementation to show how it works, I think this will help clarify things. -t
On Fri, May 22, 2009 at 11:38 AM, troy d. straszheim <troy@resophonic.com>wrote:
Beman Dawes wrote:
On Thu, May 21, 2009 at 2:40 PM, Vladimir Prus <vladimir@codesourcery.com
wrote:
troy d. straszheim wrote:
I share your worry. Probably some scm mechanism should take care of
it. I'm out of ideas for the moment.
svn:externals in SVN allow you to have master copies of headers say in
libs/thread/include
and have a copy checked out into
boost/thread
while making commits in boost/thread work, and commit into the master copy. Disk requirements are a question, though.
I also really worried about this header issue.
Volodya, thanks for the svn:externals suggestion. I'll try to learn more about svn:externals so I can form an opinion.
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional wisdom. It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project. You can end up needing to write tools to wrangle your externals. It won't be too hard to make a sample implementation to show how it works, I think this will help clarify things.
Is this a project that does project-wide merges? I can see how that might be a problem. Boost developers and release managers, OTOH, don't ever do project wide merges. Instead, everything happens on a library by library basis. I would expect merging a single library would be easier if its header files lived entirely within the library's boost-root/libs/libname tree. What am I missing? --Beman
Beman Dawes wrote:
On Fri, May 22, 2009 at 11:38 AM, troy d. straszheim <troy@resophonic.com>wrote:
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional wisdom. It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project. You can end up needing to write tools to wrangle your externals. It won't be too hard to make a sample implementation to show how it works, I think this will help clarify things.
Is this a project that does project-wide merges? I can see how that might be a problem.
Boost developers and release managers, OTOH, don't ever do project wide merges. Instead, everything happens on a library by library basis. I would expect merging a single library would be easier if its header files lived entirely within the library's boost-root/libs/libname tree.
Having libraries organized with nearby headers may be a better organization scheme and make some operations easier: agreed, we'll try it out. I'll maintain that whether or not to put these perhaps-better-organized directories on the ends of svn:externals is a different issue with long term consequences. Right now boost developers and release managers do single project switches; (modulo those who go to the trouble of using svnmerge.) Naturally that isn't an argument _for_ locking in to a path that excludes the possibility of bonafide merges, assuming there is some alternative. This would be a bit like (hypothetical) deciding on a path that somehow restricted us to using fixed size containers, just because we're getting by without them right now. Note that under the svn:externals scheme, one not only can't do a wide merge, one can't even wide-branch easily. You must: - branch the toplevel project (a commit) - foreach project that you want to modify - branch it (a commit) - change your externals and commit What should be an instantaneous local operation becomes a tedious operation involving multiple commits. Consider creating the c++0x branch in this context. Take also boost 'power users': these may be maintaining side projects, proposed libraries, and large patchsets. For instance, Joel Falcou and I are collaborating on an effort that requires - trunk (lots of spirit and proto) - our own patches to trunk - process - extension - other proposed libraries - our own stuff We'll be reporting bugs back to Hartmut and Joel, provoking and tracking changes on the trunk, reapplying various local patches that we're maintaining. The necessary primitive operation here is a merge: a real actual merge of the latest changes from trunk into our development branch. externals-wrangling is a big hindrance here. -t
On Mon, May 25, 2009 at 8:05 AM, troy d. straszheim <troy@resophonic.com>wrote:
Beman Dawes wrote:
On Fri, May 22, 2009 at 11:38 AM, troy d. straszheim <troy@resophonic.com
wrote:
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional wisdom. It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project. You can end up needing to write tools to wrangle your externals. It won't be too hard to make a sample implementation to show how it works, I think this will help clarify things.
Is this a project that does project-wide merges? I can see how that might be a problem.
Boost developers and release managers, OTOH, don't ever do project wide merges. Instead, everything happens on a library by library basis. I would expect merging a single library would be easier if its header files lived entirely within the library's boost-root/libs/libname tree.
Having libraries organized with nearby headers may be a better organization scheme and make some operations easier: agreed, we'll try it out. I'll maintain that whether or not to put these perhaps-better-organized directories on the ends of svn:externals is a different issue with long term consequences.
Right now boost developers and release managers do single project switches; (modulo those who go to the trouble of using svnmerge.)
Didn't svnmerge become obsolete when Subversion added merge tracking?
Naturally that isn't an argument _for_ locking in to a path that excludes the possibility of bonafide merges, assuming there is some alternative. This would be a bit like (hypothetical) deciding on a path that somehow restricted us to using fixed size containers, just because we're getting by without them right now.
I find "bona fide merges" and similar terms somewhat offensive. People are always claiming this or that source control system doesn't do "bona fide merges". Well, even the really crappy systems handle at least some merge scenarios without problems. Is there any SCS that handles all possible merge scenarios? I doubt it. What is the specific merge scenario that you are concerned about? Does the concern still apply to SVN 1.5 and later? Note that under the svn:externals scheme, one not only can't do a wide
merge,
Could you explain what you mean by "wide merge"?
one can't even wide-branch easily. You must:
- branch the toplevel project (a commit) - foreach project that you want to modify - branch it (a commit)
- change your externals and commit I'm trying to understand why this would be necessary. For a given library, what directory has the svn:externals property? What does the svn:externals property entry look like for a given library? Why do the externals have to be changed? Is the root problem that the svn:externals property has to be attached to a directory above the library's boost-root/libs/libname page because svn:externals properties targets don't support ".." in their paths? What might an optimal solution to keeping headers in their library tree look like that didn't use svn:externals? Perhaps something like this: * For users (i.e. anyone needing read-only access), a post-checkout step that moved the headers from their library tree to boost-root/boost. * For developers (i.e. anyone needing read/write access), a post-checkout step that created hard links in boost-root/boost. Something to think about if svn:externals can't be made to work. --Beman
On Tue, May 26, 2009 at 6:58 AM, Beman Dawes <bdawes@acm.org> wrote:
What might an optimal solution to keeping headers in their library tree look like that didn't use svn:externals? Perhaps something like this:
* For users (i.e. anyone needing read-only access), a post-checkout step that moved the headers from their library tree to boost-root/boost.
* For developers (i.e. anyone needing read/write access), a post-checkout step that created hard links in boost-root/boost.
Something to think about if svn:externals can't be made to work.
Hum... Just checked trunk/boost on my Windows XP machine: 89.5 megs in files, taking 150 megs of disk space. Nothing like svn::externals that creates copies is optimal space wise, aside from merging or other concerns. And anything that involves a post-checkout step will be less than robust. So maybe we should go back to first principles and rethink what we are trying to achieve. Hum... IIRC, Troy already posted something along those lines. That might be a more productive thread to continue with. --Beman
Beman Dawes wrote:
Boost developers and release managers, OTOH, don't ever do project wide merges. Instead, everything happens on a library by library basis. I would expect merging a single library would be easier if its header files lived entirely within the library's boost-root/libs/libname tree.
Having libraries organized with nearby headers may be a better organization scheme and make some operations easier: agreed, we'll try it out. I'll maintain that whether or not to put these perhaps-better-organized directories on the ends of svn:externals is a different issue with long term consequences.
Right now boost developers and release managers do single project switches; (modulo those who go to the trouble of using svnmerge.)
Didn't svnmerge become obsolete when Subversion added merge tracking?
I don't think the server was upgraded.
Naturally that isn't an argument _for_ locking in to a path that excludes the possibility of bonafide merges, assuming there is some alternative. This would be a bit like (hypothetical) deciding on a path that somehow restricted us to using fixed size containers, just because we're getting by without them right now.
I find "bona fide merges" and similar terms somewhat offensive. People are always claiming this or that source control system doesn't do "bona fide merges". Well, even the really crappy systems handle at least some merge scenarios without problems. Is there any SCS that handles all possible merge scenarios? I doubt it.
As I've pointed to on IRC already, merge tracking in SVN 1.5 is in some aspects more powerful than in git. So maybe it's best, indeed, to stop mixing version control system discussion with an otherwise tricky topic.
one can't even wide-branch easily. You must:
- branch the toplevel project (a commit) - foreach project that you want to modify - branch it (a commit)
- change your externals and commit
I'm trying to understand why this would be necessary. For a given library, what directory has the svn:externals property? What does the svn:externals property entry look like for a given library? Why do the externals have to be changed? Is the root problem that the svn:externals property has to be attached to a directory above the library's boost-root/libs/libname page because svn:externals properties targets don't support ".." in their paths?
From IRC discussion, it seems that Troy and I had different use of svn:externals in mind. I assumed that
trunk/boost has svn:externals for each headers directory, e.g. filesystem ../libs/filesystem/headers Therefore upon checkout, you get a complete copy of all headers in 'boost' dir. This arrangement does not create any problems for branching or merging whatsoever. What Troy had in mind (according to my understanding, which may be faulty), is that svn:externals are used on trunk/libs and pull libraries from some other location. This will indeed create additional inconvenience for merges, but it's not clear why this arrangement is desired. - Volodya
Vladimir Prus wrote:
Beman Dawes wrote:
Boost developers and release managers, OTOH, don't ever do project wide merges. Instead, everything happens on a library by library basis. I would expect merging a single library would be easier if its header files lived entirely within the library's boost-root/libs/libname tree.
Having libraries organized with nearby headers may be a better organization scheme and make some operations easier: agreed, we'll try it out. I'll maintain that whether or not to put these perhaps-better-organized directories on the ends of svn:externals is a different issue with long term consequences.
Right now boost developers and release managers do single project switches; (modulo those who go to the trouble of using svnmerge.)
Didn't svnmerge become obsolete when Subversion added merge tracking?
I don't think the server was upgraded.
It was. I'm using merge tracking in my own branch. Sebastian
Sebastian Redl wrote:
Vladimir Prus wrote:
Beman Dawes wrote:
Boost developers and release managers, OTOH, don't ever do project wide merges. Instead, everything happens on a library by library basis. I would expect merging a single library would be easier if its header files lived entirely within the library's boost-root/libs/libname tree.
Having libraries organized with nearby headers may be a better organization scheme and make some operations easier: agreed, we'll try it out. I'll maintain that whether or not to put these perhaps-better-organized directories on the ends of svn:externals is a different issue with long term consequences.
Right now boost developers and release managers do single project switches; (modulo those who go to the trouble of using svnmerge.)
Didn't svnmerge become obsolete when Subversion added merge tracking?
I don't think the server was upgraded.
It was. I'm using merge tracking in my own branch.
It is not unlikely I'm doing something wrong, but: $ svn mergeinfo https://svn.boost.org/svn/boost/trunk/libs/program_options https://svn.boost.org/svn/boost/branches/release/libs/program_options svn: Retrieval of mergeinfo unsupported by 'https://svn.boost.org/svn/boost' This message suggests that repository itself was not upgraded (if the server is 1.5 indeed). Are you using a different command? - Volodya
Vladimir Prus wrote:
Sebastian Redl wrote:
It was. I'm using merge tracking in my own branch.
It is not unlikely I'm doing something wrong, but:
$ svn mergeinfo https://svn.boost.org/svn/boost/trunk/libs/program_options https://svn.boost.org/svn/boost/branches/release/libs/program_options svn: Retrieval of mergeinfo unsupported by 'https://svn.boost.org/svn/boost'
This message suggests that repository itself was not upgraded (if the server is 1.5 indeed).
Are you using a different command?
I'm just using merge to merge the trunk into my own branch. But I've only used it once so far, so possibly the next merge won't go so well. (I can't reach the SVN server at the moment to try.) Sebastian
Sebastian Redl wrote:
Vladimir Prus wrote:
Sebastian Redl wrote:
It was. I'm using merge tracking in my own branch.
It is not unlikely I'm doing something wrong, but:
$ svn mergeinfo https://svn.boost.org/svn/boost/trunk/libs/program_options https://svn.boost.org/svn/boost/branches/release/libs/program_options svn: Retrieval of mergeinfo unsupported by 'https://svn.boost.org/svn/boost'
This message suggests that repository itself was not upgraded (if the server is 1.5 indeed).
Are you using a different command?
I'm just using merge to merge the trunk into my own branch. But I've only used it once so far, so possibly the next merge won't go so well. (I can't reach the SVN server at the moment to try.)
Hey Sebastian, I'll be interested to know how the next try goes. Might I ask how long the first merge took? -t
on Tue May 26 2009, Sebastian Redl <sebastian.redl-AT-getdesigned.at> wrote:
Vladimir Prus wrote:
Beman Dawes wrote:
Boost developers and release managers, OTOH, don't ever do project wide merges. Instead, everything happens on a library by library basis. I would expect merging a single library would be easier if its header files lived entirely within the library's boost-root/libs/libname tree.
Having libraries organized with nearby headers may be a better organization scheme and make some operations easier: agreed, we'll try it out. I'll maintain that whether or not to put these perhaps-better-organized directories on the ends of svn:externals is a different issue with long term consequences.
Right now boost developers and release managers do single project switches; (modulo those who go to the trouble of using svnmerge.)
Didn't svnmerge become obsolete when Subversion added merge tracking?
I don't think the server was upgraded.
It was. I'm using merge tracking in my own branch.
Hmm, http://svn.boost.org/svn/boost/ says otherwise. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On Tue, May 26, 2009 at 11:10 AM, Vladimir Prus <vladimir@codesourcery.com>wrote:
From IRC discussion, it seems that Troy and I had different use of svn:externals in mind. I assumed that
trunk/boost
has svn:externals for each headers directory, e.g.
filesystem ../libs/filesystem/headers
Therefore upon checkout, you get a complete copy of all headers in 'boost' dir. This arrangement does not create any problems for branching or merging whatsoever.
Interesting. Is there any problem because of the large number of svn:externals entries that would eventually end up in trunk/boost? A slight refinement from my point of view might be: filesystem ../libs/filesystem/boost/filesystem In other words, libs/filesystem/boost has exactly the same directory structure for filesystem headers as trunk/boost. That let's me put boost-root/libs/filesystem ahead of boost-root in the list of include directories for my IDE setup. That way, I'm compiling and making changes against the ../libs/filesystem/boost/filesystem headers, which is exactly what I want. I'll set up a little test repository here, and try out some scenarios.
What Troy had in mind (according to my understanding, which may be faulty), is that svn:externals are used on
trunk/libs
and pull libraries from some other location. This will indeed create additional inconvenience for merges, but it's not clear why this arrangement is desired.
I suspect there are several approaches that could be made to work, but some may be lot's easier than others. Let's try to settle on an approach that will work well for us. Thanks, --Beman
Beman Dawes wrote:
I find "bona fide merges" and similar terms somewhat offensive. People are always claiming this or that source control system doesn't do "bona fide merges". Well, even the really crappy systems handle at least some merge scenarios without problems. Is there any SCS that handles all possible merge scenarios? I doubt it.
What is the specific merge scenario that you are concerned about? Does the concern still apply to SVN 1.5 and later?
How about: Branch A: the trunk @ HEAD. Branch B: the trunk 100 commits ago, with fixes and patches sprinkled all over it, and a couple of proposed libraries added. If there are no conflicts the merge happens in ~5 seconds or less.
Note that under the svn:externals scheme, one not only can't do a wide
merge,
Could you explain what you mean by "wide merge"?
(as above)
one can't even wide-branch easily. You must:
- branch the toplevel project (a commit) - foreach project that you want to modify - branch it (a commit)
- change your externals and commit
I'm trying to understand why this would be necessary. For a given library, what directory has the svn:externals property? What does the svn:externals property entry look like for a given library? Why do the externals have to be changed? Is the root problem that the svn:externals property has to be attached to a directory above the library's boost-root/libs/libname page because svn:externals properties targets don't support ".." in their paths?
The externals: libs/serialization http://svn.boost.org/svn/boost/projects/serialization/trunk libs/flyweight http://svn.boost.org/svn/boost/projects/flyweight/trunk Of course I've just argued that you don't want to do this, but the reason you would is to be able to create subdistributions of boost and version components independently of one another. This doesn't take care of constructing your single include/ directory.
What might an optimal solution to keeping headers in their library tree look like that didn't use svn:externals? Perhaps something like this:
* For users (i.e. anyone needing read-only access), a post-checkout step that moved the headers from their library tree to boost-root/boost.
* For developers (i.e. anyone needing read/write access), a post-checkout step that created hard links in boost-root/boost.
Something to think about if svn:externals can't be made to work.
Well there is the approach that Qt has been using for many years, which Juergen pointed out. I coded up a prototype, see mails in this thread with 'qtsync' in the subject. -troy
on Tue May 26 2009, Beman Dawes <bdawes-AT-acm.org> wrote:
Didn't svnmerge become obsolete when Subversion added merge tracking?
Boost's SVN is still 1.4, i.e. no merge tracking. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
troy d. straszheim wrote:
Beman Dawes wrote:
On Thu, May 21, 2009 at 2:40 PM, Vladimir Prus <vladimir@codesourcery.com>wrote:
troy d. straszheim wrote:
I share your worry. Probably some scm mechanism should take care of it. I'm out of ideas for the moment. svn:externals in SVN allow you to have master copies of headers say in
libs/thread/include
and have a copy checked out into
boost/thread
while making commits in boost/thread work, and commit into the master copy. Disk requirements are a question, though.
I also really worried about this header issue.
Volodya, thanks for the svn:externals suggestion. I'll try to learn more about svn:externals so I can form an opinion.
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional wisdom. It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project.
Was that before relative svn:externals were available? - Volodya
Vladimir Prus wrote:
troy d. straszheim wrote:
Beman Dawes wrote:
On Thu, May 21, 2009 at 2:40 PM, Vladimir Prus <vladimir@codesourcery.com>wrote:
troy d. straszheim wrote:
I share your worry. Probably some scm mechanism should take care of it. I'm out of ideas for the moment. svn:externals in SVN allow you to have master copies of headers say in
libs/thread/include
and have a copy checked out into
boost/thread
while making commits in boost/thread work, and commit into the master copy. Disk requirements are a question, though.
I also really worried about this header issue.
Volodya, thanks for the svn:externals suggestion. I'll try to learn more about svn:externals so I can form an opinion.
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional wisdom. It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project.
Was that before relative svn:externals were available?
It was, but relative externals don't change anything. -t
troy d. straszheim wrote:
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional wisdom. It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project.
Was that before relative svn:externals were available?
It was, but relative externals don't change anything.
Then I don't really understand what merge issue you refer to. Can you clarify? - Volodya
Vladimir Prus wrote:
troy d. straszheim wrote:
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional wisdom. It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project. Was that before relative svn:externals were available?
It was, but relative externals don't change anything.
Then I don't really understand what merge issue you refer to. Can you clarify?
See my post in this thread with (svn:externals) in the title... -t
on Fri May 22 2009, "troy d. straszheim" <troy-AT-resophonic.com> wrote:
I also really worried about this header issue.
Volodya, thanks for the svn:externals suggestion. I'll try to learn more about svn:externals so I can form an opinion.
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional wisdom. It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project. You can end up needing to write tools to wrangle your externals.
Time for Git? -- Dave Abrahams BoostPro Computing http://www.boostpro.com
David Abrahams wrote:
on Fri May 22 2009, "troy d. straszheim" <troy-AT-resophonic.com> wrote:
I also really worried about this header issue.
Volodya, thanks for the svn:externals suggestion. I'll try to learn more about svn:externals so I can form an opinion.
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional wisdom. It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project. You can end up needing to write tools to wrangle your externals.
Time for Git?
Last time I've checked, it has no externals support whatsoever. - Volodya
on Wed Jun 03 2009, Vladimir Prus <vladimir-AT-codesourcery.com> wrote:
David Abrahams wrote:
on Fri May 22 2009, "troy d. straszheim" <troy-AT-resophonic.com> wrote:
I also really worried about this header issue.
Volodya, thanks for the svn:externals suggestion. I'll try to learn more about svn:externals so I can form an opinion.
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional
wisdom.
It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project. You can end up needing to write tools to wrangle your externals.
Time for Git?
Last time I've checked, it has no externals support whatsoever.
I use git-submodule all the time. Does that not count as externals support? -- Dave Abrahams BoostPro Computing http://www.boostpro.com
David Abrahams wrote:
on Wed Jun 03 2009, Vladimir Prus <vladimir-AT-codesourcery.com> wrote:
David Abrahams wrote:
on Fri May 22 2009, "troy d. straszheim" <troy-AT-resophonic.com> wrote:
I also really worried about this header issue.
Volodya, thanks for the svn:externals suggestion. I'll try to learn more about svn:externals so I can form an opinion.
I said I was "out of ideas", but that was a little disingenuous. This is certainly one of many possible implementations, but I urge caution here. We've been using svn:externals for subproject management for a while, against the conventional
wisdom.
It makes merging even harder, can turn a large difficult-to-merge project into an impossible to merge project. You can end up needing to write tools to wrangle your externals.
Time for Git?
Last time I've checked, it has no externals support whatsoever.
I use git-submodule all the time. Does that not count as externals support?
Oh, looks like it was added after I last read git manual. However, the description does not seem like it's as flexible as svn:external -- which is not required to be pinned to a specific revision. Can you use git-submodule to implement this layout: libs/ program_options/ includes/ boost/ program_options [ same content as includes above, even as 'includes' is modified] ? - Volodya
On Wednesday 03 June 2009 09:30:01 pm Vladimir Prus wrote:
Can you use git-submodule to implement this layout:
libs/ program_options/ includes/ boost/ program_options [ same content as includes above, even as 'includes' is modified]
I don't think that is something git submodules is well suited for. I doubt svn externals are any good either. That looks like something you need file system support for, aka symbolic links. Otherwise you need to run some command to ensure the directories are in sync. If you can not use the file system to ensure the content is identical, then you should select the command with some care. Build system commands, not source control system commands, seems logical. The build system need to ensure the sync of headers for the build anyway. Furthermore, build systems are good at avoiding unnecessary modifications of derived objects (target files). This is very important as build time would suffer from any simpler logic. An implicit or explicit copy rule in the build system for each header comes to mind. I know this to be a successful scheme in large build systems. Such builds generally consist of 2 passes: pass 1 - whenever needed, the build system: generate code (if you have code generation in build) copy public headers from source controlled location to the public location pass 2 - whenever needed, the build system: compile (against public location and component local headers only) link This can be performed per sub-projects (components) if you have good dependency tracking between sub-projects. Otherwise it may be wise to complete pass 1 for all sub-projects before starting any pass 2 build tasks. commands like:
bjam pass1
could be supported to update public headers without performing full build. Note also that such a build system "header copy" scheme, open some other possibilities that may be of interest as boost grows, which I do not dig into now. A file system based solution is much simpler, but I am not sure that is feasible as it probably is hard to do this in a portable way. You can do that in ClearCase as that SCM system comes with its own virtual file system for all development platforms. -- Bjørn Roald
Bjørn Roald wrote:
On Wednesday 03 June 2009 09:30:01 pm Vladimir Prus wrote:
Can you use git-submodule to implement this layout:
libs/ program_options/ includes/ boost/ program_options [ same content as includes above, even as 'includes' is modified]
I don't think that is something git submodules is well suited for. I doubt svn externals are any good either. That looks like something you need file system support for, aka symbolic links. Otherwise you need to run some command to ensure the directories are in sync. If you can not use the file system to ensure the content is identical, then you should select the command with some care.
Build system commands, not source control system commands, seems logical. The build system need to ensure the sync of headers for the build anyway. Furthermore, build systems are good at avoiding unnecessary modifications of derived objects (target files). This is very important as build time would suffer from any simpler logic. An implicit or explicit copy rule in the build system for each header comes to mind. I know this to be a successful scheme in large build systems. Such builds generally consist of 2 passes:
pass 1 - whenever needed, the build system: generate code (if you have code generation in build) copy public headers from source controlled location to the public location pass 2 - whenever needed, the build system: compile (against public location and component local headers only) link
This can be performed per sub-projects (components) if you have good dependency tracking between sub-projects. Otherwise it may be wise to complete pass 1 for all sub-projects before starting any pass 2 build tasks.
It is not really necessary in Boost.Build -- it can detect dependencies on generated files (including generated headers).
commands like:
bjam pass1
could be supported to update public headers without performing full build.
Note also that such a build system "header copy" scheme, open some other possibilities that may be of interest as boost grows, which I do not dig into now.
A file system based solution is much simpler, but I am not sure that is feasible as it probably is hard to do this in a portable way. You can do that in ClearCase as that SCM system comes with its own virtual file system for all development platforms.
I still have to figure what kind of links windows supports -- it seems it supports them, but I don't yet know how that works exactly, across everything
= XP.
- Volodya
On Wed, Jun 3, 2009 at 11:55 PM, Vladimir Prus <vladimir@codesourcery.com> wrote:
I still have to figure what kind of links windows supports -- it seems it supports them, but I don't yet know how that works exactly, across everything
= XP.
The keywords you'll want to look into are "hardlinks" and "junctions". HTH, --Michael Fawcett
While we're at it, here's the map of library interdependencies as generated by cmake and graphviz: http://sodium.resophonic.com/boost.pdf that was generated from the current release branch, thanks to some code that Doug put in a while back. A link from foo to bar means that foo needs to see bar's headers to compile. Requirements to build tests and examples are *NOT* shown here... those would be additional edges, I don't have a feel for how many. Some of the dependency information here may be out of date. -t
Hi ! On Thursday 21 May 2009, troy d. straszheim wrote:
Vladimir Prus wrote:
troy d. straszheim wrote: Maybe, we can still have single include directory while achieving the above goal if we make build process create symlinks to directories? That works on Linux, and I believe modern windows has some way to create links too. I am worried about other operating systems, though.
I share your worry. Probably some scm mechanism should take care of it. I'm out of ideas for the moment.
Maybe we should do it like Qt. They use a perl script called "sync-qt" to build the QTDIR/include tree on demand. The "include" tree then contains simple header files which just #include the "real" headers without anything else. Just clone git://gitorious.org/qt/qt.git and take a look at bin/sync-qt To see it working, just start a (shadow) build using "configure" and take a look at the generated include tree. The release packages contain this tree so end users don't need to have perl installed. I think that something like that should be possible with Boost.Build and CMake. Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Ingenieurgesellschaft für * voice: ++49 511 262926 57 ! Verkehrs- und Eisenbahnwesen mbH * fax : ++49 511 262926 99 ! Lister Straße 15 * juergen.hunold@ivembh.de ! www.ivembh.de * * Geschäftsführer: ! Sitz des Unternehmens: Hannover * Prof. Dr.-Ing. Thomas Siefer ! Amtsgericht Hannover, HRB 56965 * PD Dr.-Ing. Alfons Radtke !
Juergen Hunold wrote:
Maybe we should do it like Qt. They use a perl script called "sync-qt" to build the QTDIR/include tree on demand. The "include" tree then contains simple header files which just #include the "real" headers without anything else.
Just clone git://gitorious.org/qt/qt.git and take a look at bin/sync-qt To see it working, just start a (shadow) build using "configure" and take a look at the generated include tree.
The release packages contain this tree so end users don't need to have perl installed.
Thanks for pointing this out. I'd seen all those forwarding headers as a user of Qt but never realized they were automatically generated. I found them only mildly distracting; your compile errors are never exactly where you expect them, but then again not too far away. Clearly this is a workable system for Qt, but: why, originally? AFAICT, trolltech doesn't version their components separately; they don't provide subdistributions of qt; it is a massive package. Maybe it is just more convenient for them to have their headers next to their .cpps. Was it so that they could jury-merge (q.v. 'jury rig') by copying subdirectories around? Conclusion: should give it a try, see what happens. Details and walk-through ------------------------ % ./configure mkdir /tmp/qt/include mkdir /tmp/qt/include/QtCore (etc) header created for /tmp/qt/src/corelib/codecs/qiconvcodec_p.h (2) header created for /tmp/qt/src/corelib/codecs/qtextcodec.h (5) header created for /tmp/qt/src/corelib/codecs/qlatincodec_p.h (2) (etc) mkdir /tmp/qt/include/QtDBus mkdir /tmp/qt/include/QtDBus/private header created for /tmp/qt/src/dbus/qdbusxmlparser_p.h (2) header created for /tmp/qt/src/dbus/qdbusintegrator_p.h (2) (etc) when it starts building things, only the generated include paths are specified: Creating qmake. Please wait... g++ -c -o project.o -m64 -pipe -DQMAKE_OPENSOURCE_EDITION -I. -Igenerators -Igenerators/unix -Igenerators/win32 -Igenerators/mac -I/tmp/qt/include -I/tmp/qt/include/QtCore -I/tmp/qt/src/corelib/global -I/tmp/qt/src/script -DQT_NO_PCRE -DQT_BUILD_QMAKE -DQT_BOOTSTRAPPED -DQT_NO_TEXTCODEC -DQT_NO_UNICODETABLES -DQT_NO_COMPONENT -DQT_NO_STL -DQT_NO_COMPRESS -I/tmp/qt/mkspecs/linux-g++-64 -DHAVE_QCONFIG_CPP -DQT_NO_THREAD -DQT_NO_QOBJECT -DQT_NO_GEOM_VARIANT project.cpp Looking at the generated include directory: % ls -R include | wc -l 4894 about 5k files. A user would typically #include <QtGui/QScrollArea>, which would include: % cat include/QtGui/QScrollArea #include "qscrollarea.h" % cat include/QtGui/qscrollarea.h #include "../../src/gui/widgets/qscrollarea.h" % grep class src/gui/widgets/qscrollarea.h class QScrollAreaPrivate; class Q_GUI_EXPORT QScrollArea : public QAbstractScrollArea This is what happens in the source tree. In the distribution, include/QtGui/qscrollarea.h contains the code for QScrollArea instead of the forwarding include to the src directories. So one fewer forwarding step. Other random observations: * The bin/qtsync script is > 1k lines of perl. * For the most part the list of headers to generated is globbed; there is some kind of naming scheme for .h files that shouldn't have forwarders generated (looks like *_p.h are excluded). * In Qt, there are almost no headers that don't have a corresponding .cpp file. A different style than boost. * Perhaps one of the main advantages of this scheme is that you can keep your .h files right next to your .cpp files in your src/ directory. This way when you want to change QScrollArea, everything is nearby. * Qt was developed privately (no outside scm access) for many years. They distributed tarballs and accepted patches in the mail. (correct me if I'm wrong here). The announcement that direct access to SCM was available was only a couple weeks ago. * According to this post from a qt dev: http://lists.trolltech.com/qt4-preview-feedback/2004-12/thread00028-0.html, end users would "usually never need" syncqt. Now that their source code is open and anybody building from source will encounter it, I wonder if this will become more problematic. * Juergen tells me that this script originally used symlinks on unix and copies on windows. -t
troy d. straszheim wrote:
Juergen Hunold wrote:
Maybe we should do it like Qt.
I 'modularized' the release branch and implemented include-directory-generation with a python script. (see the post in this thread with 'qtsync' in the title) You can get a tarball of the svn:externals-free modularized boost here: http://www.gitorious.org/boost/straszheim/archive-tarball/modularized It works like this: Unpack the tarball, make a build directory, configure with cmake as usual. A python script called make_includes.py will be generated into $BUILD/bin/. Run it and a directory full of forwarding headers will be created in $BUILD/include/boost. That's the only include path you need. The source for the script is in tools/build/CMake/make_includes.py.in Generation of boost/ directory for distribution isn't done, but would be trivial. Each component's headers are in libs/*/include/ There is a dummy component called core: libs/core/include/ has the headers that didn't go anywhere else. Note: 1. if you add or remove headers you'd have to rerun this script. 2. there's an extra #include hop in every error/warning. We could probably do something with links on unix to get around this. 3. it requires python -t
troy d. straszheim wrote:
1. if you add or remove headers you'd have to rerun this script.
2. there's an extra #include hop in every error/warning. We could probably do something with links on unix to get around this.
3. it requires python
It should be possible (in bbv2, at least) to add a build system target to overcome these limitations, isn't it?
Hi ! On Wednesday 27 May 2009 12:13:38 Ilya Sokolov wrote:
troy d. straszheim wrote: 3. it requires python
It should be possible (in bbv2, at least) to add a build system target to overcome these limitations, isn't it?
Most probably. file generation is done now (think response files). And cmake does have the similar functionality. So it should be possible to handle this with the build system alone. Unless we get Boost.Python started... Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Ingenieurgesellschaft für * voice: ++49 511 262926 57 ! Verkehrs- und Eisenbahnwesen mbH * fax : ++49 511 262926 99 ! Lister Straße 15 * juergen.hunold@ivembh.de ! www.ivembh.de * * Geschäftsführer: ! Sitz des Unternehmens: Hannover * Prof. Dr.-Ing. Thomas Siefer ! Amtsgericht Hannover, HRB 56965 * PD Dr.-Ing. Alfons Radtke !
Ilya Sokolov wrote:
troy d. straszheim wrote:
1. if you add or remove headers you'd have to rerun this script.
2. there's an extra #include hop in every error/warning. We could probably do something with links on unix to get around this.
3. it requires python
It should be possible (in bbv2, at least) to add a build system target to overcome these limitations, isn't it?
Sounds reasonable to me. -t
Robert Ramey wrote:
This discussion demonstrates what's wrong with the current testing of the trunk.
Here is the way it should be done.
For each library that is changed, the library version on the trunk should be tested against the release branch. In practice this would work something like the following:
On the tester's machine start out with a current image of the release branch. For each library A switch library A's directories to the trunk run tests Restore image to release
Or (indulge me) more generally: merge branch B with release branch run tests i.e., "library A on the trunk" is just a specific case of a branch, and "switch library A's directories to the trunk" is just a specific (painfully easy) case of a merge.
This would
a) attribute failures to their proper source b) prevent testing of each library (A) with provisional (trunk) code c) guarentee that when changes are merged into the release no new failures occur. d) reduce testing time since a small expermental change in one library in the trunk wouldn't provoke rebuild and test of all its dependents.
On my own machine, I have a release branch and I've switched the serialization library directories to the trunk. I can verify that all the above mentioned benefits are in fact realized. Boost testing would benefit greatly from adopting the same strategy.
This sounds generally right to me, but I think there are problems. I'm trying to be genericize it, and I'm not sure I've got it yet: There is a 'testing' branch, to which libraries that are known to build with the release branch (with say a couple of compilers) get merged, and these libraries sit there until all-green: when that happens, they get merged to release, i.e. the testing branch essentially becomes the release. This way release *always* builds. There are other cases where two libraries are mutually dependent; This should be avoided but will happen. In this case they must go simultaneously to the testing queue. This implies that the testing branch is changing every so often. Diagram: serialization mpi parallel_graph release <= testing <= upstream <= upstream That is, a new release of serialization is in the testing queue getting hammered on by testing drones. Next up is a new version of MPI, which depends on serialization, and will have to build against whatever version of serialization goes green for release. When serialization reaches the testing queue a couple tweaks get made for borland, something majorish gets backed out. The MPI will have to update (merge) these new changes left-to-right through the queue and test privately to be sure he's ready for the testing queue. This merge operation has to be *easy* and quick. And so on. parallel_graph, in this case, is watching the testing queue and doing periodic merges and builds to be sure he's ready when it is his turn. So the "switch library A's directories" mechanism seems to break down a bit here; when serialization gets tweaked in the testing queue, MPI has to somehow "switch directories"... and so on backwards up the queue. I'm not seeing it quite yet. -t
troy d. straszheim wrote:
Robert Ramey wrote:
This discussion demonstrates what's wrong with the current testing of the trunk.
Here is the way it should be done.
For each library that is changed, the library version on the trunk should be tested against the release branch. In practice this would work something like the following:
On the tester's machine start out with a current image of the release branch. For each library A switch library A's directories to the trunk run tests Restore image to release
Or (indulge me) more generally:
merge branch B with release branch run tests
i.e., "library A on the trunk" is just a specific case of a branch, and "switch library A's directories to the trunk" is just a specific (painfully easy) case of a merge.
I don't quite agree. Assuming I have boost version N installed on my system (no matter by what means), I may want to test a given library (from whatever branch) against the installed system, not necessarily a working copy of a whole boost tree. Thus the minimal use case I'd like to be considered is to have one isolated library be built / tested, where all its prerequisites are 'outside' the tree. In that sense those prerequisites should all be treated alike, as part of the 'development environment' the current library needs. Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...
Stefan Seefeld wrote:
troy d. straszheim wrote:
Robert Ramey wrote:
This discussion demonstrates what's wrong with the current testing of the trunk.
Here is the way it should be done.
For each library that is changed, the library version on the trunk should be tested against the release branch. In practice this would work something like the following:
On the tester's machine start out with a current image of the release branch. For each library A switch library A's directories to the trunk run tests Restore image to release
Or (indulge me) more generally:
merge branch B with release branch run tests
i.e., "library A on the trunk" is just a specific case of a branch, and "switch library A's directories to the trunk" is just a specific (painfully easy) case of a merge.
I don't quite agree. Assuming I have boost version N installed on my system (no matter by what means), I may want to test a given library (from whatever branch) against the installed system, not necessarily a working copy of a whole boost tree.
Can you explain why -- as in specific use cases? Especially given the lack of source- and binary- compatibility guarantees. - Volodya
Vladimir Prus wrote:
I don't quite agree. Assuming I have boost version N installed on my system (no matter by what means), I may want to test a given library (from whatever branch) against the installed system, not necessarily a working copy of a whole boost tree.
Can you explain why -- as in specific use cases? Especially given the lack of source- and binary- compatibility guarantees.
May be that's precisely what I want to test. :-) Seriously, if I'm working on a boost library, I may want to test against different versions of boost (requisite libs). And, if some of those are already installed, why should I have to pull out an old branch from the repository just to regenerate them ? Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...
Stefan Seefeld wrote:
Vladimir Prus wrote:
I don't quite agree. Assuming I have boost version N installed on my system (no matter by what means), I may want to test a given library (from whatever branch) against the installed system, not necessarily a working copy of a whole boost tree.
Can you explain why -- as in specific use cases? Especially given the lack of source- and binary- compatibility guarantees.
May be that's precisely what I want to test. :-) Seriously, if I'm working on a boost library, I may want to test against different versions of boost (requisite libs).
Why? I assume not out of mere curiosity, but because you assume some users will use mixed Boost, and then we're back to the question why they might want to do that. It might be theoretically good for the case when library X is know to be broken in 1.N+1 and library Y is known to have some cool features, so you get 1.N of everything and 1.N+1 version of Y, but is that the only use case you have in mind? It seems that for pretty much every project, especially open source, getting a coherent set of components is considerably less risky than mixing different versions. - Volodya
Vladimir Prus wrote:
Stefan Seefeld wrote:
Vladimir Prus wrote:
I don't quite agree. Assuming I have boost version N installed on my system (no matter by what means), I may want to test a given library (from whatever branch) against the installed system, not necessarily a working copy of a whole boost tree.
Can you explain why -- as in specific use cases? Especially given the lack of source- and binary- compatibility guarantees.
May be that's precisely what I want to test. :-) Seriously, if I'm working on a boost library, I may want to test against different versions of boost (requisite libs).
Why? I assume not out of mere curiosity, but because you assume some users will use mixed Boost, and then we're back to the question why they might want to do that.
Let's not get hung up on this particular use-case. Let's just assume people want to see (or even validate) whether a particular boost component (library) builds against some other (externally managed) boost prerequisite. If you don't like that use-case, here is another one: A long time ago, Rene was working on a buildbot setup for boost. I suggested that a plausible situation would be that contributors might want to help testing boost, but not the entire thing, but only one module at a time. They would have a known-good ('golden') build of boost somewhere on their system, and a full build&test-cycle would then consist in checking out the module, build it, test it, send back results to the buildmaster. Quite a reasonably small chunk of work, we thought. The whole idea was that this was fine-grained enough such that 1) more people would be willing to contribute and 2) the time such a task would take would be reasonably small, such that the official build / test report would always reflect the current state (or with a very small delay). At the time we discussed that, the test results posted were typically between one and two weeks behind the state of the repository. Do you find this use-case more acceptable ? Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...
Stefan Seefeld wrote:
Vladimir Prus wrote:
Stefan Seefeld wrote:
Vladimir Prus wrote:
I don't quite agree. Assuming I have boost version N installed on my system (no matter by what means), I may want to test a given library (from whatever branch) against the installed system, not necessarily a working copy of a whole boost tree.
Can you explain why -- as in specific use cases? Especially given the lack of source- and binary- compatibility guarantees.
May be that's precisely what I want to test. :-) Seriously, if I'm working on a boost library, I may want to test against different versions of boost (requisite libs).
Why? I assume not out of mere curiosity, but because you assume some users will use mixed Boost, and then we're back to the question why they might want to do that.
Let's not get hung up on this particular use-case. Let's just assume people want to see (or even validate) whether a particular boost component (library) builds against some other (externally managed) boost prerequisite.
I don't like "let's just assume". We can build quite a complicated scheme that does not solve anybody's problem.
If you don't like that use-case, here is another one: A long time ago, Rene was working on a buildbot setup for boost. I suggested that a plausible situation would be that contributors might want to help testing boost, but not the entire thing, but only one module at a time. They would have a known-good ('golden') build of boost somewhere on their system, and a full build&test-cycle would then consist in checking out the module, build it, test it, send back results to the buildmaster. Quite a reasonably small chunk of work, we thought.
The whole idea was that this was fine-grained enough such that 1) more people would be willing to contribute and 2) the time such a task would take would be reasonably small, such that the official build / test report would always reflect the current state (or with a very small delay).
I think testing a subset of Boost is orthogonal to mixing different versions. You can checkout all of Boost, and run tests for select libraries. - Volodya
Vladimir Prus wrote:
Stefan Seefeld wrote:
Let's not get hung up on this particular use-case. Let's just assume people want to see (or even validate) whether a particular boost component (library) builds against some other (externally managed) boost prerequisite.
I don't like "let's just assume". We can build quite a complicated scheme that does not solve anybody's problem.
Fair enough. I just fail to see what's so complicated about this (both, the use-case as well as its implementation): All that is needed is for any given module a way to express what prerequisites it has. Doing this is fundamental to a lot of different tasks beside the above use-case (example: packaging). So, to me your suggested alternative ("let's just check out everything") sounds like you want to avoid solving the problem at hand.
I think testing a subset of Boost is orthogonal to mixing different versions. You can checkout all of Boost, and run tests for select libraries.
I didn't mean to be talking about mixing versions. All I meant to suggest is that for building and testing a given boost module we may not want to assume the rest of boost is present as a source tree, but as a separately installed boost package. Also, boost is big enough that I may just not like having to "check out all of Boost", if all I'm interested in is a single boost module. Stefan -- ...ich hab' noch einen Koffer in Berlin...
Stefan Seefeld wrote:
I didn't mean to be talking about mixing versions. All I meant to suggest is that for building and testing a given boost module we may not want to assume the rest of boost is present as a source tree, but as a separately installed boost package. Also, boost is big enough that I may just not like having to "check out all of Boost", if all I'm interested in is a single boost module.
I'd put this under the 'build system issue' heading, and (of course) it is multifaceted. CMake provides (not boost-cmake) in its distributions a configure script that you use like this: find_package(Boost 1.38.0 system filesystem serialization) and there are ways to configure how it searches (env. variables etc), and special stuff you do to use particular libraries, etc, after it has done its thing: target_link_libraries(my_executable ${Boost_FILESYSTEM}) So this you would use with any project, boost or not. Autotools could probably be made to do something similar. But: you can't just do that, because the same jam/cmake files that live in boost.process' checkout shouldn't know whether they are part of a "big" or a small boost checkout. Maybe you just if/then it up. Which implies carrying around some buildsystem infrastructure. How much? Enough to handle the build of documentation, which implies that quickbook and spirit come along? I think we just bit our own collective tail. Then there is the 'middle' case: I have a 'big' boost tree sitting on disk, built, but not installed, and I want to hack on boost.process against that library. Are we having fun yet? *Testing* some subset is I think on yet another axis. The current cmake implementation has a way to do this (you set e.g. BOOST_TEST_LIBRARIES=spirit,serialization,variant at configure time) and then only those tests are run. The main reason this functionality exists is just because it speeds up makefile generation and build times; handy for developers. It could easily be used to let e.g. the spirit guys run their own test drones. I assume boost.build already does or could do something similiar. Anyway none of this is coalescing into a grand vision, but the discussion has been productive so far: thanks everybody. Maybe about time to write some of this up. -t
troy d. straszheim wrote:
Stefan Seefeld wrote:
I didn't mean to be talking about mixing versions. All I meant to suggest is that for building and testing a given boost module we may not want to assume the rest of boost is present as a source tree, but as a separately installed boost package. Also, boost is big enough that I may just not like having to "check out all of Boost", if all I'm interested in is a single boost module.
I'd put this under the 'build system issue' heading, and (of course) it is multifaceted. CMake provides (not boost-cmake) in its distributions a configure script that you use like this:
find_package(Boost 1.38.0 system filesystem serialization)
and there are ways to configure how it searches (env. variables etc), and special stuff you do to use particular libraries, etc, after it has done its thing:
target_link_libraries(my_executable ${Boost_FILESYSTEM})
So this you would use with any project, boost or not. Autotools could probably be made to do something similar.
But:
you can't just do that, because the same jam/cmake files that live in boost.process' checkout shouldn't know whether they are part of a "big" or a small boost checkout. Maybe you just if/then it up.
Which implies carrying around some buildsystem infrastructure. How much? Enough to handle the build of documentation, which implies that quickbook and spirit come along? I think we just bit our own collective tail.
We probably can ignore doc builds for first step. But still, it appears there should be some way to mark a given library checkout as standalone, so that it does not depend on other parts of Boost checkout. But that's somewhat tricky -- in particular, that would mean that anything in top-levle Jamroot gets ignored, and any fixes or additions in tools/build/v2 will get ignored as well. And the same problem applies to CMake I believe -- since there's quite significant and growing set of local cmake definitions. So, maybe, it's better to do it otherwise: don't allow a standalone checkout of a single library, but allow, given a full checkout of Boost, to redirect some, or all libraries to some installed locations. E.g. --use-installed=python or --use-installed-except=serialization --install-root=/usr which will cause build of serialization to use installed libraries.
Then there is the 'middle' case: I have a 'big' boost tree sitting on disk, built, but not installed, and I want to hack on boost.process against that library. Are we having fun yet?
What problem is here? You just add libs/process and there you go. At least in Boost.Build land, you don't need to hack anything outside.
*Testing* some subset is I think on yet another axis. The current cmake implementation has a way to do this (you set e.g.
BOOST_TEST_LIBRARIES=spirit,serialization,variant
at configure time) and then only those tests are run. The main reason this functionality exists is just because it speeds up makefile generation and build times; handy for developers. It could easily be used to let e.g. the spirit guys run their own test drones. I assume boost.build already does or could do something similiar.
Yes, bjam libs/{spirit,serialization/test on Linux. Windows users would have to spell down each directory explicitly. - Volodya
Vladimir Prus wrote:
We probably can ignore doc builds for first step. But still, it appears there should be some way to mark a given library checkout as standalone, so that it does not depend on other parts of Boost checkout. But that's somewhat tricky -- in particular, that would mean that anything in top-levle Jamroot gets ignored, and any fixes or additions in tools/build/v2 will get ignored as well. And the same problem applies to CMake I believe -- since there's quite significant and growing set of local cmake definitions.
So, maybe, it's better to do it otherwise: don't allow a standalone checkout of a single library, but allow, given a full checkout of Boost, to redirect some, or all libraries to some installed locations. E.g.
--use-installed=python
or
--use-installed-except=serialization --install-root=/usr
which will cause build of serialization to use installed libraries.
Hm. Let's discuss. $our_projects = the libraries we're developing at the moment, say, serialization and mpi. $checkout = our development tree. Contains full boost. $install = an install of boost, say in /usr/local/boost/(lib|include) There is a buildsystem switch to enable the build in $checkout of only $our_projects. Headerfile paths: $our_projects would see e.g. the $install boost/version.hpp, and the $checkout serialization/export.hpp. For the --use-installed-except=... case, you could implement this without symlinks by simply deleting the non $our_projects headers from the $checkout when --use-installed-except is turned on, and putting $checkout/include first, $install/include second, in the include path. I think this may have certain advantages in practice... It is a little harder to get confused about which header files are the ones actually being used. For the --use-installed=... case, you'd need symlinks, you need to construct a 'forwarding' include/boost/<links>. But maybe you do anyway. (I'm not proposing any of this. Just thinking out loud.)
Then there is the 'middle' case: I have a 'big' boost tree sitting on disk, built, but not installed, and I want to hack on boost.process against that library. Are we having fun yet?
What problem is here? You just add libs/process and there you go. At least in Boost.Build land, you don't need to hack anything outside.
I meant putting the process checkout *outside* the build tree. Now I think this is a fake use case.... nevermind. -t
This might be off topic, but I'm really lost here. What I do on my system here is to set directory libs/serialization/test as the current directory. Then I invoke bjam, process.., and library_status or compiler status to generate the test report the one library I'm interested in. bjam dependency (generation or tracking) makes sure that any prerequisite libraries are rebuilt. (maybe there should be an option to re-test those prerequisite builds). Why not just apply this process to each library? It's a lot simpler than trying to maintain and "customize" a "global view" with an array of command line options and doesn't require any more work to make the libraries "more modular". Also it has the benefit what when I muckup my Jamfile.v2, It doesn't crash the whole testing setup. Robert Ramey
troy d. straszheim wrote:
Hm,
Robert Ramey wrote:
It's a lot simpler than trying to maintain and "customize" a "global view" with an array of command line options and doesn't require any more work to make the libraries "more modular".
What are you referring to?
The current boost build/bjam has some switches like "exclude serialization" or something like that. My way of doing things eliminates the need for that. Just don't apply the process to the libraries you're not interested in. Robert Ramey
-t
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
troy d. straszheim wrote:
Hm,
Robert Ramey wrote:
It's a lot simpler than trying to maintain and "customize" a "global view" with an array of command line options and doesn't require any more work to make the libraries "more modular".
What are you referring to?
The current boost build/bjam has some switches like "exclude serialization" or something like that. My way of doing things eliminates the need for that.
It does not. Those options have nothing to do with testing, and don't affect what libraries are tested and not. They are exclusively for controlling what is built and installed. - Volodya
troy d. straszheim wrote:
Robert Ramey wrote:
This discussion demonstrates what's wrong with the current testing of the trunk.
Here is the way it should be done.
For each library that is changed, the library version on the trunk should be tested against the release branch. In practice this would work something like the following:
On the tester's machine start out with a current image of the release branch. For each library A switch library A's directories to the trunk run tests Restore image to release
Or (indulge me) more generally:
merge branch B with release branch run tests
I've avoided the word "merge" as (at least in SVN) it does updates the target which changes. switch (in SVN) is a merge in this sense. For purposes of this discussion, consider "switch" equivalent to changing a hard link to another directory.
i.e., "library A on the trunk" is just a specific case of a branch, and "switch library A's directories to the trunk" is just a specific (painfully easy) case of a merge.
Basically we agree - this is an implmentation detail at this point.
This would
a) attribute failures to their proper source b) prevent testing of each library (A) with provisional (trunk) code c) guarentee that when changes are merged into the release no new failures occur. d) reduce testing time since a small expermental change in one library in the trunk wouldn't provoke rebuild and test of all its dependents.
On my own machine, I have a release branch and I've switched the serialization library directories to the trunk. I can verify that all the above mentioned benefits are in fact realized. Boost testing would benefit greatly from adopting the same strategy.
This sounds generally right to me, but I think there are problems. I'm trying to be genericize it, and I'm not sure I've got it yet:
I've found it helpful to do it on my own machine to get used to it. It's very easy to get used to. There are a couple of things to be aware of - like accidently changing something outside one's own directory - but in general, it's very easy to get used to.
There is a 'testing' branch, to which libraries that are known to build with the release branch (with say a couple of compilers) get merged, and these libraries sit there until all-green: when that happens, they get merged to release, i.e. the testing branch essentially becomes the release. This way release *always* builds.
I'm not sure what this refers to. Perhaps CMake. What I currently do is a) test on my own machine according to the above b) upload to trunk and watch the tests. c) when I feel good about it, merge to release. b) would be more useful to me if testing on the trunk were as I describe above.
There are other cases where two libraries are mutually dependent; This should be avoided but will happen. In this case they must go simultaneously to the testing queue.
I'm not seeing that. If they are two libraries that are mutually dependent, that would be a bad thing and contradicts the whole idea of a library in the first place. If its a breaking change, that's a bad thing that will happen but doesn't break the testing regimen.
This implies that the testing branch is changing every so often.
I'm not sure what "testing branch" refers to
Diagram:
serialization mpi parallel_graph release <= testing <= upstream <= upstream
That is, a new release of serialization is in the testing queue getting hammered on by testing drones. Next up is a new version of MPI, which depends on serialization, and will have to build against whatever version of serialization goes green for release. When serialization reaches the testing queue a couple tweaks get made for borland, something majorish gets backed out. The MPI will have to update (merge) these new changes left-to-right through the queue and test privately to be sure he's ready for the testing queue. This merge operation has to be *easy* and quick.
All correct - and OK as far as I'm concerned.
And so on. parallel_graph, in this case, is watching the testing queue and doing periodic merges and builds to be sure he's ready when it is his turn.
I don't see that I have to watch the queue. The only thing I have to do is watch my tests for breaking changes in dependent libraries. Which I have to do now in any case.
So the "switch library A's directories" mechanism seems to break down a bit here; when serialization gets tweaked in the testing queue, MPI has to somehow "switch directories"... and so on backwards up the queue. I'm not seeing it quite yet.
Here's the scenario. a) I make a breaking change to serialization. b) after testing in the trunk - I merge into the release queue c) MPI tests start to break. d) I get an email from the MPI authors which starts Dear $#$#%$%$&$^*%^* e) Haggling involving me, the MPI authors and perhaps the release manager result in one of the following: i) backing out changes in the serializaiton library ii) MPI authors accomodating the change. Robert Ramey
Robert Ramey wrote:
Here's the scenario.
a) I make a breaking change to serialization. b) after testing in the trunk - I merge into the release queue c) MPI tests start to break. d) I get an email from the MPI authors which starts Dear $#$#%$%$&$^*%^* e) Haggling involving me, the MPI authors and perhaps the release manager result in one of the following: i) backing out changes in the serializaiton library ii) MPI authors accomodating the change.
Robert Ramey
I suggest a slight variation on this. a) Make breaking changes to Serialization b) Testing the changed Serialization against the release branch shows that it would break MPI, so it is not merged to release, yet c) Polite conversation with MPI authors tries to convince them that the new version serves their needs even better than before (or whatever else it takes to get them to modify) d) MPI authors modify, and testing system allows you to test the combination of new Serialization and new MPI against release branch e) Tests are all green, so new versions roll in together I understand that building a test system that could accomidate this is not going to be easy, but it would minimize the need for focused testing of the release branch, and within oddities should keep release in a clean state. It also reduces the chance for - you broke my tests contentions. John
John Phillips wrote:
Robert Ramey wrote:
Here's the scenario.
a) I make a breaking change to serialization. b) after testing in the trunk - I merge into the release queue c) MPI tests start to break. d) I get an email from the MPI authors which starts Dear $#$#%$%$&$^*%^* e) Haggling involving me, the MPI authors and perhaps the release manager result in one of the following: i) backing out changes in the serializaiton library ii) MPI authors accomodating the change.
Robert Ramey
I suggest a slight variation on this.
a) Make breaking changes to Serialization b) Testing the changed Serialization against the release branch shows that it would break MPI, so it is not merged to release, yet
It didn't occur to me that this would happen. But of would under the current testing regimen where all dependencies are re-tested.
c) Polite conversation with MPI authors tries to convince them that the new version serves their needs even better than before (or whatever else it takes to get them to modify) d) MPI authors modify, and testing system allows you to test the combination of new Serialization and new MPI against release branch e) Tests are all green, so new versions roll in together I understand that building a test system that could accomidate this is not going to be easy,
I totally disagree with this - the current bjam/test software could be used without change. Only a (small?) change to the python test script would be necessary.
but it would minimize the need for focused testing of the release branch, and within oddities should keep release in a clean state. It also reduces the chance for - you broke my tests contentions.
John
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Thu, May 21, 2009 at 8:23 AM, troy d. straszheim <troy@resophonic.com> wrote:
Poking around it looks like the current release branch was created from the fixed 1.34.1 tag in oct 2007 (@ r40341). The 1.34.1 release was in turn branched from... looks like that goes back into the days of CVS, I'll consider it lost. Git maintains that the last common commit between the current trunk and release branches is > 3 years ago:
http://gitorious.org/boost/svn/commit/4914d4934fbc7368e2d43e057f6fffdac0f464...
That sounds about right. From 1.35.0 on we start each release on the previous release rather than trunk, so there is no longer any need to create a new branch from trunk for every release.
So if the trunk is really just a staging area for merges to release, does the notion of having the release branch and trunk 'in sync' at this point make any sense?
I don't think so. For individual libraries, trunk and branches/release will be sometimes or even often be in sync, but as boost grows it no longer makes sense for the entire trunk to be in sync with branches/release.
Why have just one such staging area?
In one sense, we already have multiple "staging areas" in branches for individual libraries. But trunk provides a single agreed upon place to regression test. --Beman
Beman Dawes wrote:
In one sense, we already have multiple "staging areas" in branches for individual libraries. But trunk provides a single agreed upon place to regression test.
But how meaningful are those 'regression tests' if trunk is not properly maintained ? Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...
On Thu, May 21, 2009 at 12:05 PM, Stefan Seefeld <seefeld@sympatico.ca> wrote:
Beman Dawes wrote:
In one sense, we already have multiple "staging areas" in branches for individual libraries. But trunk provides a single agreed upon place to regression test.
But how meaningful are those 'regression tests' if trunk is not properly maintained ?
I can't speak for all libraries, but for the libraries I maintain, the libraries my libraries depend on, and the libraries in trunk I use as a release manager, trunk is "properly maintained". I'm also no seeing a lot of complaints on the list about libraries on trunk not being "properly maintained". That could change over time, of course, but right now things don't look all that bad to me. --Beman
2009/5/21 troy d. straszheim <troy@resophonic.com>:
Hm, that svn command isn't very helpful and runs for ~5 minutes. I'm seeing about 160k lines total changed in actual code (.cpp, .ipp, .hpp). Full report in case anybody is interested:
I had a look through a some of these changes. A lot of them are pretty new and will probably get merged soon anyway. Some seemed to be oversights, so I filed a few tickets. A couple of libraries had some changes from last year that haven't been merged, but their regression tests weren't looking that great so they're probably not ready for release. But hopefully they'll get picked up again. Daniel
participants (13)
-
Beman Dawes -
Bjørn Roald -
Daniel James -
David Abrahams -
Ilya Sokolov -
John Phillips -
Juergen Hunold -
Michael Fawcett -
Robert Ramey -
Sebastian Redl -
Stefan Seefeld -
troy d. straszheim -
Vladimir Prus