Radical idea for repository structure
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places. * Start with the assumption that each library has a boost/ directory containing its subset of the headers it currently has. So, for example, Boost.Python would have boost/ python.hpp python/ ... where "..." above is identical to the current contents of $BOOST_ROOT/boost/python/ * Our release process would merge the boost/ directories * To test a library from a source distribution, you'd need to get the boost/ directories of any libraries it depends on into the #include path. * This list of dependency libraries would be encoded in each dependent library's Jamfile. * Presto, a way to explicitly declare and track library inter-dependencies! If you fail to declare a dependency in your Jamfile, your tests won't compile. Thoughts? -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
David Abrahams wrote:
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places.
* Start with the assumption that each library has a boost/ directory containing its subset of the headers it currently has. So, for example, Boost.Python would have
boost/ python.hpp python/ ...
where "..." above is identical to the current contents of $BOOST_ROOT/boost/python/
* Our release process would merge the boost/ directories
* To test a library from a source distribution, you'd need to get the boost/ directories of any libraries it depends on into the #include path.
* This list of dependency libraries would be encoded in each dependent library's Jamfile.
* Presto, a way to explicitly declare and track library inter-dependencies! If you fail to declare a dependency in your Jamfile, your tests won't compile.
Thoughts?
Which problems with the release process such an organization would solve? This organisation clearly requires some work to implement, so it should give some real advantage in return. - Volodya
on Mon Sep 17 2007, Vladimir Prus <ghost-AT-cs.msu.su> wrote:
David Abrahams wrote:
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places.
* Start with the assumption that each library has a boost/ directory containing its subset of the headers it currently has. So, for example, Boost.Python would have
boost/ python.hpp python/ ...
where "..." above is identical to the current contents of $BOOST_ROOT/boost/python/
* Our release process would merge the boost/ directories
* To test a library from a source distribution, you'd need to get the boost/ directories of any libraries it depends on into the #include path.
* This list of dependency libraries would be encoded in each dependent library's Jamfile.
* Presto, a way to explicitly declare and track library inter-dependencies! If you fail to declare a dependency in your Jamfile, your tests won't compile.
Thoughts?
Which problems with the release process such an organization would solve?
I wasn't suggesting it as a way to solve problems with the release process. However, having explicit dependency declarations would probably make efficient incremental testing more practical, because we'd know exactly which libraries needed to have tests run for any change.
This organisation clearly requires some work to implement, so it should give some real advantage in return.
So far it's just an idea that seems like it has potential organizational benefits. It certainly would make it easier to move in libraries from the sandbox. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
David Abrahams wrote:
on Mon Sep 17 2007, Vladimir Prus <ghost-AT-cs.msu.su> wrote:
David Abrahams wrote:
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places.
* Start with the assumption that each library has a boost/ directory containing its subset of the headers it currently has. So, for example, Boost.Python would have
boost/ python.hpp python/ ...
where "..." above is identical to the current contents of $BOOST_ROOT/boost/python/
* Our release process would merge the boost/ directories
* To test a library from a source distribution, you'd need to get the boost/ directories of any libraries it depends on into the #include path.
* This list of dependency libraries would be encoded in each dependent library's Jamfile.
* Presto, a way to explicitly declare and track library inter-dependencies! If you fail to declare a dependency in your Jamfile, your tests won't compile.
Thoughts?
Which problems with the release process such an organization would solve?
I wasn't suggesting it as a way to solve problems with the release process. However, having explicit dependency declarations would probably make efficient incremental testing more practical, because we'd know exactly which libraries needed to have tests run for any change.
"Probably"? The only problem with incremental testing now I know are stale results, which is quite separate.
This organisation clearly requires some work to implement, so it should give some real advantage in return.
So far it's just an idea that seems like it has potential organizational benefits. It certainly would make it easier to move in libraries from the sandbox.
For me, it seems like there's zillion ways to organise SVN, which were discussed for about month. That's good, but I don't see SVN organization is the particular important thing, given that many projects are doing just fine with relatively straight-forward organization. I worry that yet another round of "how to place files in SVN" discussion, will distract from real problems. - Volodya
on Tue Sep 18 2007, Vladimir Prus <ghost-AT-cs.msu.su> wrote:
I wasn't suggesting it as a way to solve problems with the release process. However, having explicit dependency declarations would probably make efficient incremental testing more practical, because we'd know exactly which libraries needed to have tests run for any change.
"Probably"?
Uh, yeah. Sarcasm isn't really necessary here.
The only problem with incremental testing now I know are stale results, which is quite separate.
The time it takes to run the tests is always a factor. Representing and evaluating all the dependencies in the project each time is expensive.
I worry that yet another round of "how to place files in SVN" discussion, will distract from real problems.
Fair enough. I'm not claiming this is urgent or even necessarily a problem-solver, so if there are more important things to discuss, please raise them and re-focus our attention. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
On 9/19/07, David Abrahams <dave@boost-consulting.com> wrote:
on Tue Sep 18 2007, Vladimir Prus <ghost-AT-cs.msu.su> wrote:
[snip]
I worry that yet another round of "how to place files in SVN" discussion, will distract from real problems.
Fair enough. I'm not claiming this is urgent or even necessarily a problem-solver, so if there are more important things to discuss, please raise them and re-focus our attention.
IMHO, the modularity David is proposing, which, btw, I asked something like it twice before with regard to include directories being exclusive to libraries, but being mostly ignored, is a very important concern. This modularity would easier patching the trunk and seeing what broke what. The dependencies could also, latter, be given by boost.build. For that, though, even the header-only libraries should have their headers separate, forcing every dependency to be explicit in the Jamfiles. And, until now, I can't foresee any future problems, but maybe I'm being too optimistic, I dont know. Problems with CMake, maybe?
-- Dave Abrahams Boost Consulting http://www.boost-consulting.com
Best regards, -- Felipe Magno de Almeida
Felipe Magno de Almeida wrote:
On 9/19/07, David Abrahams <dave@boost-consulting.com> wrote:
on Tue Sep 18 2007, Vladimir Prus <ghost-AT-cs.msu.su> wrote:
[snip]
I worry that yet another round of "how to place files in SVN" discussion, will distract from real problems. Fair enough. I'm not claiming this is urgent or even necessarily a problem-solver, so if there are more important things to discuss, please raise them and re-focus our attention.
IMHO, the modularity David is proposing, which, btw, I asked something like it twice before with regard to include directories being exclusive to libraries, but being mostly ignored, is a very important concern. This modularity would easier patching the trunk and seeing what broke what. The dependencies could also, latter, be given by boost.build. For that, though, even the header-only libraries should have their headers separate, forcing every dependency to be explicit in the Jamfiles. And, until now, I can't foresee any future problems, but maybe I'm being too optimistic, I dont know. Problems with CMake, maybe?
I'd second that! I think this is a *very* important matter to discuss and IMO in no way is it distracting from our discussions. Regards, -- Joel de Guzman http://www.boost-consulting.com http://spirit.sf.net
David Abrahams wrote:
on Tue Sep 18 2007, Vladimir Prus <ghost-AT-cs.msu.su> wrote:
I wasn't suggesting it as a way to solve problems with the release process. However, having explicit dependency declarations would probably make efficient incremental testing more practical, because we'd know exactly which libraries needed to have tests run for any change.
"Probably"?
Uh, yeah. Sarcasm isn't really necessary here.
There's no sarcasm. "Probably" also implies "not sure", and I naturally wonder whether you indeed unsure the proposed change will help with incremental testing.
The only problem with incremental testing now I know are stale results, which is quite separate.
The time it takes to run the tests is always a factor. Representing and evaluating all the dependencies in the project each time is expensive.
Unless you test a given library against pre-build pre-installed Boost, I don't see how this organization cuts down on any resources. And testing against pre-build pre-installed Boost is independent from any directory structure in SVN.
I worry that yet another round of "how to place files in SVN" discussion, will distract from real problems.
Fair enough. I'm not claiming this is urgent or even necessarily a problem-solver, so if there are more important things to discuss, please raise them and re-focus our attention.
Ok, for a start: 1. No release manager(s) is appointed. 2. No schedule, and no list of goals is published for 1.35. - Volodya
on Wed Sep 19 2007, Vladimir Prus <ghost-AT-cs.msu.su> wrote:
David Abrahams wrote:
on Tue Sep 18 2007, Vladimir Prus <ghost-AT-cs.msu.su> wrote:
I wasn't suggesting it as a way to solve problems with the release process. However, having explicit dependency declarations would probably make efficient incremental testing more practical, because we'd know exactly which libraries needed to have tests run for any change.
"Probably"?
Uh, yeah. Sarcasm isn't really necessary here.
There's no sarcasm. "Probably" also implies "not sure", and I naturally wonder whether you indeed unsure the proposed change will help with incremental testing.
As with any other change, you don't know for sure until you try it.
The only problem with incremental testing now I know are stale results, which is quite separate.
The time it takes to run the tests is always a factor. Representing and evaluating all the dependencies in the project each time is expensive.
Unless you test a given library against pre-build pre-installed Boost, I don't see how this organization cuts down on any resources.
I don't understand what this has to do with pre-built/installed Boost. Given an explicit representation of library dependencies we can, based on the directories containing changed files, quickly compute all the libraries whose tests need to be run and avoid evaluating the Jamfiles of other libraries.
I worry that yet another round of "how to place files in SVN" discussion, will distract from real problems.
Fair enough. I'm not claiming this is urgent or even necessarily a problem-solver, so if there are more important things to discuss, please raise them and re-focus our attention.
Ok, for a start:
1. No release manager(s) is appointed. 2. No schedule, and no list of goals is published for 1.35.
Please start new threads for these things if you want them to get attention. Mostly because Beman demanded it after we settled the (not yet published) release procedure, I am going to take ultimate responsibility for management of the release, with help from Doug Gregor and Jeff Garland and probably a few other people. We had at least one very promising volunteer to whom I haven't had a chance to respond (if you're listening, we're interested -- sorry about the silence). I'm going to try to post the plan Real Soon Now, but at the moment other things are pressing. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
On 2007-09-19, David Abrahams <dave@boost-consulting.com> wrote:
Given an explicit representation of library dependencies we can, based on the directories containing changed files, quickly compute all the libraries whose tests need to be run and avoid evaluating the Jamfiles of other libraries.
Could we possibly use some dependency scanner, e.g. like in scons, that dynamically finds dependencies rather than requiring that they be explicit? If I understand, the explicit dependency statements you're talking about are really already expressed in the header-inclusion tree. Since the tests also (I assume) include the headers of the things they test, determining which tests to run after you make some changes is really just a question of including the tests themselves in the dependency calculations. That is, make the running of the tests contingent upon their needing to be rebuilt (due to either dependency changes or actual test changes). The main benefit of this approach, of course, is that the decision to run a test is fully automated and (assuming the scanner is correct) thorough. There is no risk that the explicit dependency statements are over- or under-specified. The downsides are that the scanning may take time (though not much, in my experience) and that you have to write a (potentially quite) complex scanner. It would also, of course, complicate the build system to some degree. -- Austin Bingham
on Tue Sep 25 2007, Austin Bingham <abingham-AT-arlut.utexas.edu> wrote:
On 2007-09-19, David Abrahams <dave@boost-consulting.com> wrote:
Given an explicit representation of library dependencies we can, based on the directories containing changed files, quickly compute all the libraries whose tests need to be run and avoid evaluating the Jamfiles of other libraries.
Could we possibly use some dependency scanner, e.g. like in scons, that dynamically finds dependencies rather than requiring that they be explicit?
We already have one built into bjam. Explicit dependency notation has some advantages. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
I'm using boost-build build to build my own code. I experimented with a structure that was similar to what you're describing. It eventually evolved into my current system which still uses boost-build, but defines no libraries. Instead, my code is split into units of much smaller size. A unit is a single cpp or hpp file. In the jamfile, each unit lists its dependencies, so that when you need the functionality provided by a given unit, it'll automatically pull in whatever else is needed. Of course the release process works the way you describe, except that it has the same file-level granularity. I am not sure if this would also work for Boost. Emil Dotchevski http://www.revergestudios.com/reblog/index.php?n=ReCode David Abrahams wrote:
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places.
* Start with the assumption that each library has a boost/ directory containing its subset of the headers it currently has. So, for example, Boost.Python would have
boost/ python.hpp python/ ...
where "..." above is identical to the current contents of $BOOST_ROOT/boost/python/
* Our release process would merge the boost/ directories
* To test a library from a source distribution, you'd need to get the boost/ directories of any libraries it depends on into the #include path.
* This list of dependency libraries would be encoded in each dependent library's Jamfile.
* Presto, a way to explicitly declare and track library inter-dependencies! If you fail to declare a dependency in your Jamfile, your tests won't compile.
Thoughts?
On 9/17/07, David Abrahams <dave@boost-consulting.com> wrote:
[snip]
Thoughts?
That is how I have my projects organized. It is extremely more modular and easier, and is completely supported by boost.build.
-- Dave Abrahams Boost Consulting http://www.boost-consulting.com
The Astoria Seminar ==> http://www.astoriaseminar.com
-- Felipe Magno de Almeida
David Abrahams wrote:
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places.
* Start with the assumption that each library has a boost/ directory containing its subset of the headers it currently has. So, for example, Boost.Python would have
boost/ python.hpp python/ ...
where "..." above is identical to the current contents of $BOOST_ROOT/boost/python/
* Our release process would merge the boost/ directories
* To test a library from a source distribution, you'd need to get the boost/ directories of any libraries it depends on into the #include path.
* This list of dependency libraries would be encoded in each dependent library's Jamfile.
* Presto, a way to explicitly declare and track library inter-dependencies! If you fail to declare a dependency in your Jamfile, your tests won't compile.
Thoughts?
Question 1: Does that mean different Jamfiles are needed for releases vs SVN working copies? That is too error prone - a way would have to be found so that the same Jamfile works regardless of whether it is run against a release or a SVN working copy. Question 2: Let's say lib A depends on lib B, and B in turn depends on lib C. Does that mean lib A's Jamfile must list both B and C? If such transitive dependencies have to be listed, that's a problem because if C gets changed to depend on lib D, then the Jamfile for A, B, and C all have to be changed, and that's a mess. OTOH, if only direct dependencies have to be listed in the Jamfile, it might possibly work. Question 3: What about developers who use an IDE or traditional make files? For example, using the VC++ IDE, the "Additional Include Directories" property is currently set to $(BOOST_ROOT) and you are done. On IDE's where this sort of property can be inherited, you only have to specify that in one place, and all Boost compiles work correctly. With your scheme it sounds like every library used will require an additional include path, and that path will be different for release compared to SVN working copies. That gets old fast. Or have you figured out a simpler way? Curious, --Beman
On 9/17/07, Beman Dawes <bdawes@acm.org> wrote:
David Abrahams wrote:
[snip]
Thoughts?
Question 1: Does that mean different Jamfiles are needed for releases vs SVN working copies? That is too error prone - a way would have to be found so that the same Jamfile works regardless of whether it is run against a release or a SVN working copy.
We can have only the SVN directory structure. People can have a complete include directory with bjam install. Which could copy headers from all libraries there.
Question 2: Let's say lib A depends on lib B, and B in turn depends on lib C. Does that mean lib A's Jamfile must list both B and C? If such transitive dependencies have to be listed, that's a problem because if C gets changed to depend on lib D, then the Jamfile for A, B, and C all have to be changed, and that's a mess. OTOH, if only direct dependencies have to be listed in the Jamfile, it might possibly work.
It is transitive. No need to type dependencies of dependencies in the jamfiles.
Question 3: What about developers who use an IDE or traditional make files? For example, using the VC++ IDE, the "Additional Include Directories" property is currently set to $(BOOST_ROOT) and you are done. On IDE's where this sort of property can be inherited, you only have to specify that in one place, and all Boost compiles work correctly. With your scheme it sounds like every library used will require an additional include path, and that path will be different for release compared to SVN working copies. That gets old fast. Or have you figured out a simpler way?
Just type bjam install and use it.
Curious,
--Beman
Best regards, -- Felipe Magno de Almeida
on Mon Sep 17 2007, Beman Dawes <bdawes-AT-acm.org> wrote:
David Abrahams wrote:
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places.
* Start with the assumption that each library has a boost/ directory containing its subset of the headers it currently has. So, for example, Boost.Python would have
boost/ python.hpp python/ ...
where "..." above is identical to the current contents of $BOOST_ROOT/boost/python/
* Our release process would merge the boost/ directories
* To test a library from a source distribution, you'd need to get the boost/ directories of any libraries it depends on into the #include path.
* This list of dependency libraries would be encoded in each dependent library's Jamfile.
* Presto, a way to explicitly declare and track library inter-dependencies! If you fail to declare a dependency in your Jamfile, your tests won't compile.
Thoughts?
Question 1: Does that mean different Jamfiles are needed for releases vs SVN working copies? That is too error prone - a way would have to be found so that the same Jamfile works regardless of whether it is run against a release or a SVN working copy.
I'm certain that's easy to arrange.
Question 2: Let's say lib A depends on lib B, and B in turn depends on lib C. Does that mean lib A's Jamfile must list both B and C?
We get to decide.
If such transitive dependencies have to be listed, that's a problem because if C gets changed to depend on lib D, then the Jamfile for A, B, and C all have to be changed, and that's a mess. OTOH, if only direct dependencies have to be listed in the Jamfile, it might possibly work.
Having to specify everything is more explicit, but more labor-intensive. It's a tradeoff. I'm not sure which one is right, but I think probably the implicit way is explicit enough.
Question 3: What about developers who use an IDE or traditional make files? For example, using the VC++ IDE, the "Additional Include Directories" property is currently set to $(BOOST_ROOT) and you are done. On IDE's where this sort of property can be inherited, you only have to specify that in one place, and all Boost compiles work correctly. With your scheme it sounds like every library used will require an additional include path, and that path will be different for release compared to SVN working copies. That gets old fast. Or have you figured out a simpler way?
If you install the headers you can use them with a single directory in the #include path. It's also possible to use svn:externals to automate the collection of subdirectories so you can do this directly in an svn working copy. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
David Abrahams wrote:
It's also possible to use svn:externals to automate the collection of subdirectories so you can do this directly in an svn working copy.
But note that svn externals are broken in a lot of ways. It is not possible to do a commit across externals, for example. Ditto for diffs. Just a caution that svn externals are not the panacea they are made out to be. -Dave
on Mon Sep 17 2007, "David A. Greene" <greened-AT-obbligato.org> wrote:
David Abrahams wrote:
It's also possible to use svn:externals to automate the collection of subdirectories so you can do this directly in an svn working copy.
But note that svn externals are broken in a lot of ways. It is not possible to do a commit across externals, for example. Ditto for diffs.
What do you mean "a commit across externals?" I certainly have had no trouble with them.
Just a caution that svn externals are not the panacea they are made out to be.
We've had plenty of those warnings already, but I'm not claiming they're a panacea. I'm only claiming that if you want a directory in your SVN working development copy that has the exact structure of a Boost installation's include directory, svn:externals makes it possible. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
David Abrahams wrote:
Just a caution that svn externals are not the panacea they are made out to be.
We've had plenty of those warnings already, but I'm not claiming they're a panacea. I'm only claiming that if you want a directory in your SVN working development copy that has the exact structure of a Boost installation's include directory, svn:externals makes it possible.
And how will you handle header files directly in 'boost' dir? svn:externals only work on directories. - Volodya
Hi ! On Dienstag 18 September 2007, Vladimir Prus wrote:
David Abrahams wrote:
Just a caution that svn externals are not the panacea they are made out to be.
We've had plenty of those warnings already, but I'm not claiming they're a panacea. I'm only claiming that if you want a directory in your SVN working development copy that has the exact structure of a Boost installation's include directory, svn:externals makes it possible.
And how will you handle header files directly in 'boost' dir? svn:externals only work on directories.
Perhaps we can use the Qt4 approach. Qt4 has a general "include" directory which contains script-generated forward header which simply #include the real header using relativ paths: Example: QTDIR/include/QtCore/qstring.h contains: #include "../../src/corelib/tools/qstring.h" and everything works. Trolltech used symlinks in Qt3 where available and moved the headers on Win32 using some P4 magic, but I think this was a maintenance nightmare. The current solution is better. It would be nice if Boost.Build could then automagically create this forwards instead of using a script... Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Ingenieurgesellschaft für * voice: ++49 511 262926 57 ! Verkehrs- und Eisenbahnwesen mbH * fax : ++49 511 262926 99 ! Lister Straße 15 * juergen.hunold@ivembh.de ! www.ivembh.de * * Geschäftsführer: ! Sitz des Unternehmens: Hannover * Prof. Dr.-Ing. Thomas Siefer ! Amtsgericht Hannover, HRB 56965 * PD Dr.-Ing. Alfons Radtke !
on Tue Sep 18 2007, Vladimir Prus <ghost-AT-cs.msu.su> wrote:
David Abrahams wrote:
Just a caution that svn externals are not the panacea they are made out to be.
We've had plenty of those warnings already, but I'm not claiming they're a panacea. I'm only claiming that if you want a directory in your SVN working development copy that has the exact structure of a Boost installation's include directory, svn:externals makes it possible.
And how will you handle header files directly in 'boost' dir? svn:externals only work on directories.
Yes, the common 'boost' directory contains the externals. Header files directly in boost are changed to be simple forwarding headers to the real files, which live in the subdirectories for particular libraries. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
On Tuesday 18 September 2007 09:31, David Abrahams wrote:
But note that svn externals are broken in a lot of ways. It is not possible to do a commit across externals, for example. Ditto for diffs.
What do you mean "a commit across externals?" I certainly have had no trouble with them.
Let's say I have repository foo with some externals that looks like this: foo/ bar/ (external) baz/ (external) src/ module1/ module2/ If I make changes to files in module1 and files in baz and want to commit them, I must do two different commits, one for module1 and one for baz. This can become troublesome if these commits are dependent as there will be a window where the repository is not buildable/runnable/whatever. I also cannot get a single diff file of all the changes. I have to run two separate diffs. These aren't necessarily showstoppers, but it's very inconvenient at times.
Just a caution that svn externals are not the panacea they are made out to be.
We've had plenty of those warnings already, but I'm not claiming they're a panacea. I'm only claiming that if you want a directory in your SVN working development copy that has the exact structure of a Boost installation's include directory, svn:externals makes it possible.
I know that you didn't claim externals are a panacea, but many, many people do. They seem to think externals works like CVS did when you did a checkout of a repository under a working copy of a different repository and could do diffs, commits, etc. and have CVS "do the right thing." Externals does not work that way. Curiously, emacs svn mode allows you to do diffs across externals (presumably running separate diffs and aggregating the results), but isn't smart enough to do commits across externals. -Dave
David A. Greene wrote:
Let's say I have repository foo with some externals that looks like this:
foo/ bar/ (external) baz/ (external) src/ module1/ module2/
If I make changes to files in module1 and files in baz and want to commit them, I must do two different commits, one for module1 and one for baz. This can become troublesome if these commits are dependent as there will be a window where the repository is not buildable/runnable/whatever.
I also cannot get a single diff file of all the changes. I have to run two separate diffs.
These aren't necessarily showstoppers, but it's very inconvenient at times.
I think what Dave imagines looks more like libs/ serialization/ spirit/ phoenix/ regex/ ... aggregates/ boost/ -> serialization/boost -> spirit/boost -> phoenix/boost libs/ -> serialization/libs -> spirit/libs -> phoenix/libs i.e. the externals are merely for convenience of checking out and updating when you want to run the SVN version of boost, not for actual development, certainly not for ever having commits. The question is whether that works - can externals be "merged" this way?
Curiously, emacs svn mode allows you to do diffs across externals (presumably running separate diffs and aggregating the results), but isn't smart enough to do commits across externals.
Or not foolhardy enough to do it. The race condition you pointed out above would be enough to make developers think twice about providing this as an automated feature. Sebastian Redl
David Abrahams wrote:
on Mon Sep 17 2007, Beman Dawes <bdawes-AT-acm.org> wrote:
David Abrahams wrote:
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places.
* Start with the assumption that each library has a boost/ directory containing its subset of the headers it currently has. So, for example, Boost.Python would have
boost/ python.hpp python/ ...
where "..." above is identical to the current contents of $BOOST_ROOT/boost/python/
* Our release process would merge the boost/ directories
* To test a library from a source distribution, you'd need to get the boost/ directories of any libraries it depends on into the #include path.
* This list of dependency libraries would be encoded in each dependent library's Jamfile.
* Presto, a way to explicitly declare and track library inter-dependencies! If you fail to declare a dependency in your Jamfile, your tests won't compile.
Thoughts?
I would guess that almost everyone believes that it would be best if the header files for each library actually lived in that library. Lots of people have suggested we move to that organization. The showstopper has always been to avoid requiring end-users and library developers to have to many paths to their build system (bjam, IDE, makefile, whatever). So if I understand you correctly, you are suggesting that as long as we can still provide end-users of releases with a single path (by copying to boost-root/boost during release preparation), it might be OK to force developers to do have to specify a path for each library their library depends on. That strikes me as OK if they only have to specify direct dependencies, which is something they control. But it isn't OK if they have to specify indirect dependencies, which out of their control, much harder to identify, change all the time, and are sometimes subject exactly what macros are defined for a given compile.
Question 1: Does that mean different Jamfiles are needed for releases vs SVN working copies? That is too error prone - a way would have to be found so that the same Jamfile works regardless of whether it is run against a release or a SVN working copy.
I'm certain that's easy to arrange.
Good. That overcomes one hurdle.
Question 2: Let's say lib A depends on lib B, and B in turn depends on lib C. Does that mean lib A's Jamfile must list both B and C?
We get to decide.
See above. I don't see how it can work if a Jamfile has to be changed every time its indirect dependencies change.
If such transitive dependencies have to be listed, that's a problem because if C gets changed to depend on lib D, then the Jamfile for A, B, and C all have to be changed, and that's a mess. OTOH, if only direct dependencies have to be listed in the Jamfile, it might possibly work.
Having to specify everything is more explicit, but more labor-intensive. It's a tradeoff. I'm not sure which one is right, but I think probably the implicit way is explicit enough.
I think that is the only practical approach, if the Jamfiles actually have to work against a raw svn checkout (which I think they should).
Question 3: What about developers who use an IDE or traditional make files? For example, using the VC++ IDE, the "Additional Include Directories" property is currently set to $(BOOST_ROOT) and you are done. On IDE's where this sort of property can be inherited, you only have to specify that in one place, and all Boost compiles work correctly. With your scheme it sounds like every library used will require an additional include path, and that path will be different for release compared to SVN working copies. That gets old fast. Or have you figured out a simpler way?
If you install the headers you can use them with a single directory in the #include path.
Hum... I've always rejected that approach as too cumbersome for developers. But nowadays disk space is so cheap I have separate boost working copies, one I update regularly and one I keep more stable to use for development. It wouldn't be such a problem if I was having to update the development working copy via a script that did the header install. We also might want to brainstorm ways that avoid developers having to do a header install after every svn up. Possibilities: * Hard links. Say the header install created hard links rather than copies. Then there is no need to do another header install after an update unless a header is added or deleted. * Hard links, maintained by svn and thus totally transparent. Not sure if this is possible, but it sure is attractive. I'll research this one. * Provide a property file for popular IDE's like Visual Studio. Not as attractive, but might be a fallback if other approaches fail.
It's also possible to use svn:externals to automate the collection of subdirectories so you can do this directly in an svn working copy.
David Greene's posting looks worrysome for externals. --Beman
Beman Dawes wrote:
I would guess that almost everyone believes that it would be best if the header files for each library actually lived in that library. Lots of people have suggested we move to that organization. The showstopper has always been to avoid requiring end-users and library developers to have to many paths to their build system (bjam, IDE, makefile, whatever).
So if I understand you correctly, you are suggesting that as long as we can still provide end-users of releases with a single path (by copying to boost-root/boost during release preparation), it might be OK to force developers to do have to specify a path for each library their library depends on.
That strikes me as OK if they only have to specify direct dependencies, which is something they control. But it isn't OK if they have to specify indirect dependencies, which out of their control, much harder to identify, change all the time, and are sometimes subject exactly what macros are defined for a given compile.
Unless I'm missing your point, this is exactly what boost.build/bjam are good at. Developers specify the requirements and usage-requirements for their project and/or rules. Developer of library A only needs to list the project id and rule for dependency B. The developer of library A clearly knows that they are using dependency B. Any usage-requirements for library B are described in the Jamfile for library B. Developer A never needs to track or worry about them. <snip>
See above. I don't see how it can work if a Jamfile has to be changed every time its indirect dependencies change.
The only time Jamfiles should be getting changed is if one of your direct requirements or the usage-requirements of your library changes. Indirect dependencies would be controlled by the usage-requirements of the other libraries creating them. -- ---------------------------------- Michael Caisse Object Modeling Designs www.objectmodelingdesigns.com
"David Abrahams" <dave@boost-consulting.com> wrote in message news:87myvl143d.fsf@grogan.peloton...
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places.
* Start with the assumption that each library has a boost/ directory containing its subset of the headers it currently has. So, for example, Boost.Python would have
boost/ python.hpp python/ ...
where "..." above is identical to the current contents of $BOOST_ROOT/boost/python/
* Our release process would merge the boost/ directories
* To test a library from a source distribution, you'd need to get the boost/ directories of any libraries it depends on into the #include path.
* This list of dependency libraries would be encoded in each dependent library's Jamfile.
* Presto, a way to explicitly declare and track library inter-dependencies! If you fail to declare a dependency in your Jamfile, your tests won't compile.
Thoughts?
This is exactly an idea I pitched some time ago with explicit library dependencies specification. I believe it is a way to go. Gennadiy
on Mon Sep 17 2007, "Gennadiy Rozental" <rogeeff-AT-gmail.com> wrote:
This is exactly an idea I pitched some time ago with explicit library dependencies specification.
Sorry I didn't pay more attention at the time. Good idea, though ;-) -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
David Abrahams wrote:
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places.
<snip>
Thoughts?
Sounds like a great idea to me. I've worked on some fairly large projects and keeping libraries and components in their own subtrees always makes build and maintenance life easier. As for my own projects, I do something very similar (but instead of 'boost' it is 'omd'). I have "#include "omd/something_amazing.h" and then let my boost-build project ids, usage-requirements, and dependencies take care of the rest. (I sure love Boost.Build). Libraries are then separated to a traditional subversion layout with each library having a trunk/tags/branches. I don't have a slick build script to pull a distribution together. I am typically wanting my_amazing_library and any dependencies, so I just copy them to a client's sandbox and go from there. It would be wonderful to do this with Boost also... especially as the Boost offering grows. Right now I just distribute the whole thing for ease, but it would be nice to just divide out what I need. Using your proposed structure will provide that dependency list. -- ---------------------------------- Michael Caisse Object Modeling Designs www.objectmodelingdesigns.com
Hello, David! You wrote on Mon, 17 Sep 2007 15:31:18 -0400: DA> A few days ago I started thinking about the implications of giving DA> each Boost library a separate subtree of the repository, and it led me DA> to some interesting places. I think about this every time I see how you fight with current release ;) Reorganazing boost structure to separate libs is very very good idea, but require more work and rethinking, if you really want to get usefull results. When I faced up with same king of problem in my organization where I work, I came up with this: SVN structure repository boost.config 1.0 .... 1.1 .... trunk build include boost config src doc tests boost.filesystem 1.0 ... 2.0 .... trunk build include boost filesystem src doc test Developer structure C:\Work same as SVN structure, but without unneeded library versions + Jamfile project-root.jam boost-build.jam
From user perspective boost will be used as monolithic library
#incude <boost/config.hpp> #include <boost/filesystem/path.hpp> because of usage requirments in boost.config and boost.filesystem Jamfiles. The developer Jamfile should looks like this: use-project /boost/config : boost.config/build//boost_config : <version>1.0 ; use-project /boost/config : boost.config/build//boost_config : <version>1.1 ; use-project /boost/filesystem : boost.config/build//boost_filesystem : <version>1.0 ; use-project /boost/filesystem: boost.config/build//boost_filesystem : <version>2.0 ; This require extending boost.build to understand such constructs but its not evil its good. Such modification to use-project rule will give ability to skip parsing not used libraries Jamfiles on early stage. Having notion of external/internal dependencies will allow to build projects such as A(1.0) -internaly depends - B(1.0)-internaly depends-D(1.0) |----externaly depends C(1.0)-internaly depends-D(1.1) But external/internal dependencies is very (IMO) dangerous beast. Without good so/dll analyzer tool it will be too far error prone, but anyway it will be usefull. What about IDE users? Having such structure will kill all at once :) Without a tool that will generate IDEs project files this structure will force users do a lot more work setting up dependencies, addition include directories and so on. I have wrote such a tool and it was able to parse simple Jamfile and generate VC 8.0 project and solution. But now I am on the way of total refactoring :) Thoughts? With best regards, Konstantin Litvinenko.
on Tue Sep 18 2007, "Konstantin Litvinenko" <Konstantin.Litvinenko-AT-malva.ua> wrote:
What about IDE users? Having such structure will kill all at once :) Without a tool that will generate IDEs project files this structure will force users do a lot more work setting up dependencies, addition include directories and so on.
Again, svn:externals can help
I have wrote such a tool and it was able to parse simple Jamfile and generate VC 8.0 project and solution. But now I am on the way of total refactoring :)
Thoughts?
It's the right result but probably the wrong approach. Boost.Build itself should generate the project/solution/makefile/whatever. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
David Abrahams wrote:
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places.
* Start with the assumption that each library has a boost/ directory containing its subset of the headers it currently has. So, for example, Boost.Python would have
boost/ python.hpp python/ ...
where "..." above is identical to the current contents of $BOOST_ROOT/boost/python/
* Our release process would merge the boost/ directories
* To test a library from a source distribution, you'd need to get the boost/ directories of any libraries it depends on into the #include path.
Right. A fallout of this is that it all in a sudden becomes easy to test a development snapshot of a single library against all prerequisite libraries being from some stable state. Yay for modularity. Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...
David Abrahams wrote:
A few days ago I started thinking about the implications of giving each Boost library a separate subtree of the repository, and it led me to some interesting places.
[snip] That would be ideal for Spirit having its own release cycle and with each release having a miniboost copy of only those libraries needed. Regards, -- Joel de Guzman http://www.boost-consulting.com http://spirit.sf.net
David Abrahams wrote:
* Presto, a way to explicitly declare and track library inter-dependencies! If you fail to declare a dependency in your Jamfile, your tests won't compile.
The build system is _not_ bjam based, but I use similar setup at work. It is very flexible and the fact that you simply can maintain and enforce a none-cyclic graph between what we call packages make your life a lot simpler. One important point that you need to take care of is that tests for a library need separate packages(Jamfile) from the library itself. Tests typically depend on the library being tested, but also test framework and possibly other things the lib does not need. This should never become an explicit dependency of the lib. I guess this may mean that header only libraries somehow need a Jamfile to indicate its immediate dependencies. Another useful point that has not been mentioned in other replies is that such a dependency graph allow controlled build/test concurrency at the package level. -- Bjørn
participants (15)
-
Austin Bingham -
Beman Dawes -
Bjørn Roald -
David A. Greene -
David Abrahams -
Emil Dotchevski -
Felipe Magno de Almeida -
Gennadiy Rozental -
Joel de Guzman -
Juergen Hunold -
Konstantin Litvinenko -
Michael Caisse -
Sebastian Redl -
Stefan Seefeld -
Vladimir Prus