[modularization] proposal and poll
Dear Boost mailing list members, I would like to revive the topic of Boost inter-library dependencies, suggest two global changes to Boost and ask whether you agree with the course of action I propose. RATIONALE Boost recently transitioned from a monolithic SVN repository to a "modular" Git repository. One of the potential benefits of this new structure is that it may allow users to copy much less of Boost when they need only a small part of it. I believe that two additional, evolutionary changes are needed to truly unlock that benefit. The currently-standard way to use the new Git repository is to recursively clone the super-project and all its submodules, one for each of the >100 libraries. In order to use Boost in a truly modular way, a developer would rather just clone only the libraries she needs (including dependencies). Unfortunately, there is currently no feasible way to do that [see also 1]: - Manually: this is cumbersome and generally hard because there are many (undocumented) interdependencies between Boost libraries. - Automatically: as far as I know Ryppl is the only existing attempt at providing automated dependency handling for Boost, but it is a very ambitious project that attempts to do much more and it is currently neither finished nor being actively developed. I think there could be a less ambitious but faster way to provide automated dependency handling. It would take no more than some static configuration, to list for each library what other Boost libraries it depends on, combined with a simple script that automatically traverses the dependency graph and clones just the necessary submodules. This script may then be hooked into the bootstrap script. I think I could write the dependency handler, and I would be willing to, though I can't currently provide any indication of when I would be able to start. However, even if we had an automated way to handle dependencies, that would not be a sufficient solution to true modularity. As has been discussed before, the interdependencies between Boost libraries are currently so strong, that even using a single header file may likely require the user to copy a substantial portion of Boost [2]. This is especially the case because there are many cyclical dependencies [3, 4]. Stephen Kelly investigated how the interdependencies between Boost libraries could be reduced, especially the cyclical dependencies. He concluded that most cyclical dependencies could be eliminated and that the dependency graph could be simplified substantially overall. He also described concrete ways to achieve this. [3 and onwards] I believe it would still be worthwhile to reduce interdependencies between Boost libraries in the way Stephen proposed, or with a very similar approach. Stephen told me he is willing to re-analyze the dependencies as many times as needed as he did before and to make recommendations. Together with an automated way to handle dependencies, this should enable users to install only a small portion of Boost if that is all they need. It will also make the Boost super- project more flexible overall. PROPOSAL The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order: 1. Reduction of dependencies between Boost libraries. 2. Simple but effective automation of dependency handling. POLL I think it may be easier for the steering committee to justify any decision on this proposal, if they have hard statistics about the amount of support it receives. Therefore, I invite every opinionated person reading this to reply with a vote, either in favour of the proposal or against it. I invite you to do this even if you do not feel the need or find the time to provide arguments for your position. You can later re-vote if you change your mind. I will count the votes, and when there are more than 30 I will start posting the statistics occasionally. Votes are public so everyone can verify my data when necessary. Anyone, including the steering committee, can then use the numbers however they choose. Thanks in advance, -Julian ____________ [1] http://lists.boost.org/Archives/boost/2014/05/213115.php [2] http://lists.boost.org/Archives/boost/2013/10/207379.php [3] http://lists.boost.org/Archives/boost/2013/10/207384.php [4] http://lists.boost.org/Archives/boost/2013/10/207422.php [5] http://lists.boost.org/Archives/boost/2014/05/213229.php
On 29 May 2014 at 22:22, Julian Gonggrijp wrote:
I would like to revive the topic of Boost inter-library dependencies, suggest two global changes to Boost and ask whether you agree with the course of action I propose.
Here goes the fun again. Glad I'm not the one starting a thread this time!
I think there could be a less ambitious but faster way to provide automated dependency handling. It would take no more than some static configuration, to list for each library what other Boost libraries it depends on, combined with a simple script that automatically traverses the dependency graph and clones just the necessary submodules. This script may then be hooked into the bootstrap script. I think I could write the dependency handler, and I would be willing to, though I can't currently provide any indication of when I would be able to start.
If a fork of Boost becomes necessary later this year, I was thinking an accurate dependency graph could be generated by having the unit test CI reuse fabricate's (https://code.google.com/p/fabricate/) compiler file loading tracing implementation. That would fire the dependency graph into a future AFIO based graph store, and forked Boost would download the appropriate graph shard on demand into a local graphstore. That would enable easy per-library source distros, and other such goodies.
However, even if we had an automated way to handle dependencies, that would not be a sufficient solution to true modularity. As has been discussed before, the interdependencies between Boost libraries are currently so strong, that even using a single header file may likely require the user to copy a substantial portion of Boost [2]. This is especially the case because there are many cyclical dependencies [3, 4].
Stephen Kelly investigated how the interdependencies between Boost libraries could be reduced, especially the cyclical dependencies. He concluded that most cyclical dependencies could be eliminated and that the dependency graph could be simplified substantially overall. He also described concrete ways to achieve this. [3 and onwards]
The hard part isn't what to do. The hard part is gaining community buy in when "their" code gets broken by the necessary changes. I have concluded it isn't possible, hence me proposing a fork where such "radical" changes won't upset anyone. Like the Python 3.x vs 2.x fork.
I think it may be easier for the steering committee to justify any decision on this proposal, if they have hard statistics about the amount of support it receives. Therefore, I invite every opinionated person reading this to reply with a vote, either in favour of the proposal or against it. I invite you to do this even if you do not feel the need or find the time to provide arguments for your position. You can later re-vote if you change your mind.
I will count the votes, and when there are more than 30 I will start posting the statistics occasionally. Votes are public so everyone can verify my data when necessary. Anyone, including the steering committee, can then use the numbers however they choose.
Decisive and immediate action on this is vital if Boost is to survive. I vote yes to both, ASAP. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On Thu, May 29, 2014 at 4:22 PM, Julian Gonggrijp <j.gonggrijp@gmail.com> wrote:
I think there could be a less ambitious but faster way to provide automated dependency handling. It would take no more than some static configuration, to list for each library what other Boost libraries it depends on, combined with a simple script that automatically traverses the dependency graph and clones just the necessary submodules. This script may then be hooked into the bootstrap script.
I love this idea.
PROPOSAL
The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:
1. Reduction of dependencies between Boost libraries. 2. Simple but effective automation of dependency handling.
I'd like to suggest reversing this order: describing even the existing dependencies would be valuable. It would also give us a way to quantify incremental reductions in dependencies. "Look, this pull request removes two dependencies from this library and seven of its consumers!"
POLL
I invite every opinionated person reading this to reply with a vote, either in favour of the proposal or against it.
I am thrilled that you are championing this idea. Please proceed!
On 2014-05-29 16:22, Julian Gonggrijp wrote:
PROPOSAL
The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:
1. Reduction of dependencies between Boost libraries. 2. Simple but effective automation of dependency handling.
I very much applaud all efforts on 1., but I'm slightly against 2. insofar as it reduces the motivation to work on 1. (Read: If by 2. you mean the tracking of dependencies, I think this is very useful in support of the overall goal. In contrast, if you mean tools to clone files automatically, that would provide incentives to continue carelessly adding unneeded dependencies, so I'd vote against it.) Good luck ! Stefan -- ...ich hab' noch einen Koffer in Berlin...
Stefan Seefeld wrote:
On 2014-05-29 16:22, Julian Gonggrijp wrote:
1. Reduction of dependencies between Boost libraries. 2. Simple but effective automation of dependency handling.
I very much applaud all efforts on 1., but I'm slightly against 2. insofar as it reduces the motivation to work on 1.
(Read: If by 2. you mean the tracking of dependencies, I think this is very useful in support of the overall goal. In contrast, if you mean tools to clone files automatically, that would provide incentives to continue carelessly adding unneeded dependencies, so I'd vote against it.)
By point 2. I meant a script that looks up what other submodules a particular Boost submodule depends on, and checks out those submodules in order to provide the user with a local copy (which can then be installed). I believe this would be a disincentive for library authors to add more dependencies, because library interdependencies become more marked in this way (currently the dependencies are invisible because users always download all of Boost). Nat Goodspeed suggested a similar positive effect for point 2, where the existence of a file that documents the dependencies of a library may motivate people to reduce the number of dependencies. Would you vote in favour of that? Either way, I'll count your vote in favour of point 1.
Good luck !
Thank you, and thanks to the others who voted so far. -Julian
On Thu, May 29, 2014 at 1:22 PM, Julian Gonggrijp wrote:
- Manually: this is cumbersome and generally hard because there are many (undocumented) interdependencies between Boost libraries. - Automatically: as far as I know Ryppl is the only existing attempt at providing automated dependency handling for Boost, but it is a very ambitious project that attempts to do much more and it is currently neither finished nor being actively developed.
I think there could be a less ambitious but faster way to provide automated dependency handling. It would take no more than some static configuration, to list for each library what other Boost libraries it depends on, combined with a simple script that automatically traverses the dependency graph and clones just the necessary submodules. This script may then be hooked into the bootstrap script. I think I could write the dependency handler, and I would be willing to, though I can't currently provide any indication of when I would be able to start.
It would be nice if both approaches (static configuration, automatic identification) take into account - or allow you to specify - conditional dependencies (e.g. dependencies based on identification of supported or unsupported features determined by Boost.Config). For example: Boost.Align should only be considered as depending on Boost.TypeTraits (for boost::alignment_of) if BOOST_NO_CXX11_HDR_TYPE_TRAITS is defined; otherwise it uses the C++11 standard library (for std::alignment_of). Similarly, it only depends on Boost.Utility (for boost::addressof) if BOOST_NO_CXX11_STD_ADDRESSOF is defined; otherwise it uses the C++11 standard library (for std::addressof). For anyone wanting to consume Boost.Align, should they have a conforming C++11 implementation, they should see Boost.Align's dependencies as only Boost.Config and Boost.Integer. Otherwise, someone consuming Boost.Align with a totally non-conforming C++11 implementation should see its dependencies as: Boost.Config, Boost.Integer, Boost.Utility, Boost.TypeTraits. Glen
Glen Fernandes wrote:
On Thu, May 29, 2014 at 1:22 PM, Julian Gonggrijp wrote:
- Manually: this is cumbersome and generally hard because there are many (undocumented) interdependencies between Boost libraries. - Automatically: as far as I know Ryppl is the only existing attempt at providing automated dependency handling for Boost, but it is a very ambitious project that attempts to do much more and it is currently neither finished nor being actively developed.
I think there could be a less ambitious but faster way to provide automated dependency handling. It would take no more than some static configuration, to list for each library what other Boost libraries it depends on, combined with a simple script that automatically traverses the dependency graph and clones just the necessary submodules. This script may then be hooked into the bootstrap script. I think I could write the dependency handler, and I would be willing to, though I can't currently provide any indication of when I would be able to start.
It would be nice if both approaches (static configuration, automatic identification) take into account - or allow you to specify - conditional dependencies (e.g. dependencies based on identification of supported or unsupported features determined by Boost.Config).
In my reply to Robert Ramey just now I mentioned the possibility to use Boost.Bjam's <toolset> to handle this. There might be other ways as well. Given that it is probably feasible to account for this, would you vote in favour of what I propose?
For example:
Boost.Align should only be considered as depending on Boost.TypeTraits (for boost::alignment_of) if BOOST_NO_CXX11_HDR_TYPE_TRAITS is defined; otherwise it uses the C++11 standard library (for std::alignment_of). Similarly, it only depends on Boost.Utility (for boost::addressof) if BOOST_NO_CXX11_STD_ADDRESSOF is defined; otherwise it uses the C++11 standard library (for std::addressof).
For anyone wanting to consume Boost.Align, should they have a conforming C++11 implementation, they should see Boost.Align's dependencies as only Boost.Config and Boost.Integer. Otherwise, someone consuming Boost.Align with a totally non-conforming C++11 implementation should see its dependencies as: Boost.Config, Boost.Integer, Boost.Utility, Boost.TypeTraits.
Glen
On Thu, May 29, 2014 at 10:20 PM, Julian Gonggrijp wrote:
Glen Fernandes wrote:
It would be nice if both approaches (static configuration, automatic identification) take into account - or allow you to specify - conditional dependencies (e.g. dependencies based on identification of supported or unsupported features determined by Boost.Config).
In my reply to Robert Ramey just now I mentioned the possibility to use Boost.Bjam's <toolset> to handle this. There might be other ways as well. Given that it is probably feasible to account for this, would you vote in favour of what I propose?
If the result is tooling that can address the following: - Developer can conveniently express "I care about Boost libraries L1, L2, L3 for compilers C1, C2, C3" and the tooling replies with: "Get L1, L2, L3, L9, L18, L27". - If the developer instead expresses "I care about Boost libraries L1, L2, L3 but only for compiler C1" and the tooling replies with: "Get L1, L2, L3, L9". Then sure: I think that would be useful to have. (Whether it is implemented by adopting either of the two approaches). Glen
Glen Fernandes wrote:
It would be nice if both approaches (static configuration, automatic identification) take into account - or allow you to specify - conditional dependencies (e.g. dependencies based on identification of supported or unsupported features determined by Boost.Config).
It would be nice, but is it implementable? How could the dependency tracker determine what Boost.Config will define?
On Fri, May 30, 2014 at 8:31 AM, Peter Dimov <lists@pdimov.com> wrote:
Glen Fernandes wrote:
It would be nice if both approaches (static configuration, automatic identification) take into account - or allow you to specify - conditional dependencies (e.g. dependencies based on identification of supported or unsupported features determined by Boost.Config).
It would be nice, but is it implementable? How could the dependency tracker determine what Boost.Config will define?
I've been assuming that Julian's proposal will require, at least initially, that someone (the library maintainer, an interested volunteer) manually edit a specification in some syntax of the other libraries on which this one directly depends. That information could be informed by, but need not be identical to, the output of a tool. In that case, Glen's suggestion only requires that the specification syntax include some way of annotating each dependency: "only for the following toolsets..." It may also be possible to write a (wave-based?) tool that analyzes the preprocessor conditions under which each #include is reached, but I see that as very much a separable (and deferrable) effort.
Nat Goodspeed wrote:
On Fri, May 30, 2014 at 8:31 AM, Peter Dimov <lists@pdimov.com> wrote:
Glen Fernandes wrote:
It would be nice if both approaches (static configuration, automatic identification) take into account - or allow you to specify - conditional dependencies (e.g. dependencies based on identification of supported or unsupported features determined by Boost.Config).
It would be nice, but is it implementable? How could the dependency tracker determine what Boost.Config will define?
I've been assuming that Julian's proposal will require, at least initially, that someone (the library maintainer, an interested volunteer) manually edit a specification in some syntax of the other libraries on which this one directly depends. That information could be informed by, but need not be identical to, the output of a tool.
In that case, Glen's suggestion only requires that the specification syntax include some way of annotating each dependency: "only for the following toolsets..."
It may also be possible to write a (wave-based?) tool that analyzes the preprocessor conditions under which each #include is reached, but I see that as very much a separable (and deferrable) effort.
The initial topic of this thread was 'Does Boost have modularization as a goal'. There seems to have been topic-creep. Nevertheless, I suggest you and anyone else interested read http://www.cmake.org/cmake/help/git-master/manual/cmake-compile-features.7.h... which I implemented, and which allows transitive propagation of compile feature requirements at a buildsystem level. Especially if you think 'Does this library use C++11' and 'Does this library use C++14' are good questions, you should reconsider and re-frame your question in terms of what features are required from the compiler instead. Especially consider that in the light of what Robert Ramey wrote about 4 variants of C++. There are not 4 variants. There are many more than 4. There are as many variants as there are versions of compilers you support, which each support different features. No MSVC release 'supports C++11' (or C++98, if you want to be very pedantic), but it does support some features described in it which other compilers also support. That is what is relevant, not the C++ standard version: http://www.kdab.com/modern-cmake-with-qt-and-boost/#compile-feature-specific... CMake is aware of all C++11 and C++14 features and can depend on a target or not depending on whether the compiler in use supports a particular feature or features: http://www.cmake.org/cmake/help/git-master/prop_gbl/CMAKE_CXX_KNOWN_FEATURES... though someone would have to contribute and maintain a listing for MSVC supported features something like: https://gitorious.org/cmake/steveires-cmake/source/0156b7f4:Modules/Compiler... Even if you don't rely on CMake, you should consider a similar design with b2 or any new tool you create. Trying to keep the information in Boost.Config and extract it for use by the buildsystem or dependency resolver would be awkward. You can move the information about what compiler has what feature into b2 and use that information to generate Boost.Config, as CMake does: http://www.cmake.org/cmake/help/git-master/module/WriteCompilerDetectionHead... HOWEVER: All of this is off-topic for this thread. I'm listing it only because people are thinking too far down the line and getting stuck. I recommend dropping discussion of tooling and how to record/resolve dependencies. Understand that there are solutions for what will come up down the line and you don't need to worry about them now. I recommend you return to the question of whether Boost wants to modularize or not. If you do want that, then consider whether my recommendations linked in the start of the thread are specific or concrete. There were suggestions in this thread that my recommendations were not concrete or specific or didn't show any benefit. I don't understand or agree with those conclusions, but I think those are the useful and small questions you guys should focus on if you want to make any conclusive progress at all on this issue. Thanks, Steve.
Stephen Kelly wrote:
I recommend you return to the question of whether Boost wants to modularize or not.
The root of the problem is that absence of modularity is (a) invisible, and (b) carries no cost. It is invisible because there is no current easy way for developers and users to see a dependency report. It carries no cost because currently Boost developers pay no price at all for introducing a dependency and derive no benefit from severing one. (Realistically, neither do most users.) Unless and until these two issues are addressed, modularization will - in my opinion - not happen. I am not entirely sure what needs to be done on the second front, but we can at least do something on the first one.
Op 30 mei 2014 om 16:33 heeft "Peter Dimov" <lists@pdimov.com> het volgende geschreven:
Stephen Kelly wrote:
I recommend you return to the question of whether Boost wants to modularize or not.
The root of the problem is that absence of modularity is (a) invisible, and (b) carries no cost.
It is invisible because there is no current easy way for developers and users to see a dependency report.
100% agree here.
It carries no cost because currently Boost developers pay no price at all for introducing a dependency and derive no benefit from severing one. (Realistically, neither do most users.)
Everyone except for the maintainer pays a price for additional dependencies. Hence everyone except for the maintainer also benefits from fewer dependencies. This is an issue that has been raised before. I do agree that having visible dependencies would increase the incentive for maintainers to reduce those dependencies. Making the dependencies visible is an intended side effect of part 2 of my plan. -Julian
On Fri, May 30, 2014 at 9:45 AM, Stephen Kelly <hello@steveire.com> wrote:
The initial topic of this thread was 'Does Boost have modularization as a goal'. There seems to have been topic-creep.
Well, Julian asked a specific question, and he started a new thread in which to do it (albeit related to, and even linked to, some previous threads). I think his approach was appropriate. That said, I admit I'm being a bit pedantic here.
Especially if you think 'Does this library use C++11' and 'Does this library use C++14' are good questions, you should reconsider and re-frame your question in terms of what features are required from the compiler instead.
Okay, excellent point.
HOWEVER: All of this is off-topic for this thread. I'm listing it only because people are thinking too far down the line and getting stuck. I recommend dropping discussion of tooling and how to record/resolve dependencies.
I recommend you return to the question of whether Boost wants to modularize or not.
I have been taking the liberty of inferring that each positive response to Julian's question is a +1 for the overarching question of whether Boost should finish modularizing. Any responder who supports Julian's proposal but not the concept of modular Boost, please feel free to correct me.
If you do want that, then consider whether my recommendations linked in the start of the thread are specific or concrete.
Steven, you specifically mean this post, yes? http://lists.boost.org/Archives/boost/2013/10/207384.php And the recommendations in question are to move a handful of specific header files to specific other libraries, correct? n.b. Although I hadn't previously seen the linked post (I joined the dev list rather recently), it directly supports my claim a few messages back that _tracking_ cross-library dependencies immediately gives us metrics by which to evaluate all subsequent dependency-reduction efforts. :-)
Nat Goodspeed wrote:
On Fri, May 30, 2014 at 11:03 AM, Nat Goodspeed <nat@lindenlab.com> wrote:
Steven, you specifically mean this post, yes?
Apologies, Stephen! My eye obviously caught on your e-mail address rather than your real name. :-(
Yes, sorry. I know it's confusing that I sign-off as 'Steve'.
Nat Goodspeed wrote:
If you do want that, then consider whether my recommendations linked in the start of the thread are specific or concrete.
Steven, you specifically mean this post, yes? http://lists.boost.org/Archives/boost/2013/10/207384.php
I posted a follow-up too: http://thread.gmane.org/gmane.comp.lib.boost.devel/245078/focus=245806 Many respondents in this thread would benefit from reading the links Julian posted I think. Many have not done so it seems.
Stephen Kelly wrote:
Nat Goodspeed wrote:
If you do want that, then consider whether my recommendations linked in the start of the thread are specific or concrete.
Steven, you specifically mean this post, yes? http://lists.boost.org/Archives/boost/2013/10/207384.php
I posted a follow-up too:
http://thread.gmane.org/gmane.comp.lib.boost.devel/245078/focus=245806
Many respondents in this thread would benefit from reading the links Julian posted I think. Many have not done so it seems.
Here's a summary of Stephen's proposed changes from the above post: 1. Move enable_if from utility to type_traits 2. Next, move boost/detail/workaround.hpp to the config library/repo [done] 3. Next, move boost/limits.hpp to the config library/repo [done] 4. Next, move boost/detail/iterator.hpp and boost/iterator/iterator_traits.hpp to the type_traits library/repo 5. Next, move boost/iterator.hpp from iterator to the utility library/repo 6. Move boost/version.hpp to config [done] 7. Next, move boost/pointee.hpp from iterator to detail 8. Move exception/detail/attribute_noreturn.hpp into config 9. Move boost/throw_exception.hpp and exception.hpp into utility 10. Move parts of {vector_,}property_map into graph_parallel
Stephen Kelly wrote:
[...]
HOWEVER: All of this is off-topic for this thread. I'm listing it only because people are thinking too far down the line and getting stuck. I recommend dropping discussion of tooling and how to record/resolve dependencies. Understand that there are solutions for what will come up down the line and you don't need to worry about them now.
I recommend you return to the question of whether Boost wants to modularize or not.
I agree with this point. Thank you for this remark, Stephen.
If you do want that, then consider whether my recommendations linked in the start of the thread are specific or concrete. There were suggestions in this thread that my recommendations were not concrete or specific or didn't show any benefit.
I don't understand or agree with those conclusions, but I think those are the useful and small questions you guys should focus on if you want to make any conclusive progress at all on this issue.
Nat Goodspeed wrote:
I've been assuming that Julian's proposal will require, at least initially, that someone (the library maintainer, an interested volunteer) manually edit a specification in some syntax of the other libraries on which this one directly depends. That information could be informed by, but need not be identical to, the output of a tool.
In that case, Glen's suggestion only requires that the specification syntax include some way of annotating each dependency: "only for the following toolsets..."
Ah. I misunderstood. Manually specifying dependencies is probably better than nothing. It has two significant drawbacks though. First, it's easy to get them wrong. I've been surprised a number of times by boostdep.exe when it uncovered a dependency of which I wasn't aware. Second, even if you get them right at first, it's easy for them to become wrong. Adding an #include to fix a bug is something that few of us think about twice. This will be especially relevant for libraries maintained by the CMT, although I don't intend to single these out; ordinary maintenance is bound to introduce dependencies by mistake for the same reason it's easy for humans to get dependencies wrong in the first place. And even if the library is not touched at all, a header can migrate from module A (marked a dependency) to module B (not marked as such). For these reasons, manual specification of dependencies can be reliable only if these specifications are tested as part of the regression tests. That is, if the tests for module X operate in an environment in which only the stated (and transitive) dependencies of X are present.
For these reasons, manual specification of dependencies can be reliable only if these specifications are tested as part of the regression tests. That is, if the tests for module X operate in an environment in which only the stated (and transitive) dependencies of X are present.
This looked impractical to me at first, but on second thought, it probably can be done with just a modification to "b2 headers" - there's no need to actually delete the libraries not listed as dependencies, just their header links.
On Fri, May 30, 2014 at 2:14 PM, Peter Dimov <lists@pdimov.com> wrote:
For these reasons, manual specification of dependencies can be reliable only if these specifications are tested as part of the regression tests. That is, if the tests for module X operate in an environment in which only the stated (and transitive) dependencies of X are present.
Yes! The regression tests should definitely use the specification to build/test the library. This might make the full regression run take longer (if there are libs that have dependencies on non-header-only libs, they will end up getting built more than once). On the other hand, once the regression tests get setup to test modular libraries, we can break up the monolitic regression runs into much more manageable pieces (each library + dependencies)...each of which will run much faster than the whole. This will allow us to get to a more CI like regression setup where we can test libs as changes are applied. Right now most of the libraries don't get changed but maybe once a month, but they all get built/tested with each regression run. This causes the regression runs to take > 8hrs on some toolsets. This means that (on runners with multiple toolsets, like teeks99-02, 03, 05) we can go many days before a new change gets tested for a specific toolset. Tom
Yes! The regression tests should definitely use the specification to build/test the library.
This might make the full regression run take longer (if there are libs that have dependencies on non-header-only libs, they will end up getting built more than once). On the other hand, once the regression tests get setup to test modular libraries, we can break up the monolitic regression runs into much more manageable pieces (each library + dependencies)...each of which will run much faster than the whole.
That would certainly be great, but what are we tracking as dependencies here - the headers? Or everything needed to test the lib? The latter is likely to be a much bigger list. John.
Peter Dimov wrote:
Nat Goodspeed wrote:
Manually specifying dependencies is probably better than nothing. It has two significant drawbacks though. First, it's easy to get them wrong. I've been surprised a number of times by boostdep.exe when it uncovered a dependency of which I wasn't aware.
Second, even if you get them right at first, it's easy for them to become wrong. Adding an #include to fix a bug is something that few of us think about twice. This will be especially relevant for libraries maintained by the CMT, although I don't intend to single these out; ordinary maintenance is bound to introduce dependencies by mistake for the same reason it's easy for humans to get dependencies wrong in the first place.
And even if the library is not touched at all, a header can migrate from module A (marked a dependency) to module B (not marked as such).
For these reasons, manual specification of dependencies can be reliable only if these specifications are tested as part of the regression tests. That is, if the tests for module X operate in an environment in which only the stated (and transitive) dependencies of X are present.
I don't think the dependencies need to be listed manually *necessarily*, even initially, because Stephen Kelly also generated his (module-level) dependency graphs automatically if I'm right. It is true that I didn't rule out the possibility, however. You raise a valid concern. Testing whether all necessary dependencies are included was part of my plan, but I didn't mention it yet because I considered it a detail that could be covered after we agree that modularization should be finished. -Julian
Peter Dimov wrote:
Glen Fernandes wrote:
It would be nice if both approaches (static configuration, automatic identification) take into account - or allow you to specify - conditional dependencies (e.g. dependencies based on identification of supported or unsupported features determined by Boost.Config).
It would be nice, but is it implementable? How could the dependency tracker determine what Boost.Config will define?
Tool detection using the <toolset> from Boost.Jam, perhaps combined with some manual configuration, although it seems that Peter Dimov's dependency traker might be sufficient. -Julian
On Thursday 29 May 2014 22:22:05 Julian Gonggrijp wrote:
PROPOSAL
The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:
1. Reduction of dependencies between Boost libraries.
Agree, to a reasonable point. I don't think that solid libraries should be torn apart or unrelated components of different libraries should be mixed together just to reduce dependencies. The cases where this would be beneficial should be discussed with library maintainers. I think that notion of optional dependencies it also needed to achieve this goal. There are multiple places in Boost where dependencies are intentionally loosened but formally exist.
2. Simple but effective automation of dependency handling.
Taking into account your reply to Stefan, I agree that a tool for selectively downloading a subset of Boost is needed.
On 30/05/2014 10:44, Andrey Semashev wrote:
On Thursday 29 May 2014 22:22:05 Julian Gonggrijp wrote:
PROPOSAL
The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:
1. Reduction of dependencies between Boost libraries.
This is a worthy goal, I don't think anyone reasonably disagrees with this.
Agree, to a reasonable point. I don't think that solid libraries should be torn apart or unrelated components of different libraries should be mixed together just to reduce dependencies. The cases where this would be beneficial should be discussed with library maintainers.
I think that notion of optional dependencies it also needed to achieve this goal. There are multiple places in Boost where dependencies are intentionally loosened but formally exist.
Right the devil is in the detail. For example the Math lib is one of those with cyclic dependencies - there's a small core that a few libraries depend on, but which Math also uses to some degree (like lexical_cast). The question is: * How many sub libraries does it need to be split into? FP classification is one, constants another, possibly rounding functions one more, then one more for everything else. Grief. * How can such a split be done and yet still provide a seemless experience for the user - i.e. I'm strongly in favour of keeping the docs all in one place, and the header directory structure as it is now, while possibly allowing users that are just using say lexical_cast to not have to download the whole Math lib. * Ideally I'd like to keep maintenance overhead as low as possible - the change to the new Git directory structure has been quite disruptive, and everyone involved in maintaining the Math lib has got into "header hell" more than once by editing the wrong file. Splitting the lib makes that even worse. We also need sensible policies for dealing with optional components - a good example would be libraries that provide serialization support in a separate optional header. The library as such does not require Boost.Serialization, but quite rightly the optional "bridging" support is there. I asked about this last time this topic came up, but I saw no good answer? Please note: *I still think reducing dependencies is a good idea*, but I'd really like to see some kind of workflow that addresses these issues, otherwise I fear actually making things worse if libraries get harder to maintain at the alter of reduced dependencies. Regards, John.
John Maddock wrote:
We also need sensible policies for dealing with optional components - a good example would be libraries that provide serialization support in a separate optional header. The library as such does not require Boost.Serialization, but quite rightly the optional "bridging" support is there. I asked about this last time this topic came up, but I saw no good answer?
I'm not sure whether this is actually a significant problem in practice. An automated dependency tracker will be confused, but that would simply be a matter of marking up the bridge header as "do-not-track" in some way, wouldn't it?
Peter Dimov wrote:
John Maddock wrote:
We also need sensible policies for dealing with optional components - a good example would be libraries that provide serialization support in a separate optional header. The library as such does not require Boost.Serialization, but quite rightly the optional "bridging" support is there. I asked about this last time this topic came up, but I saw no good answer?
I'm not sure whether this is actually a significant problem in practice. An automated dependency tracker will be confused, but that would simply be a matter of marking up the bridge header as "do-not-track" in some way, wouldn't it?
I agree it probably won't be a big problem. -Julian
On 05/30/2014 01:17 PM, John Maddock wrote:
On 30/05/2014 10:44, Andrey Semashev wrote:
On Thursday 29 May 2014 22:22:05 Julian Gonggrijp wrote:
PROPOSAL
The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:
1. Reduction of dependencies between Boost libraries.
This is a worthy goal, I don't think anyone reasonably disagrees with this.
+1
Agree, to a reasonable point. I don't think that solid libraries should be torn apart or unrelated components of different libraries should be mixed together just to reduce dependencies. The cases where this would be beneficial should be discussed with library maintainers.
I think that notion of optional dependencies it also needed to achieve this goal. There are multiple places in Boost where dependencies are intentionally loosened but formally exist.
Right the devil is in the detail.
For example the Math lib is one of those with cyclic dependencies - there's a small core that a few libraries depend on, but which Math also uses to some degree (like lexical_cast).
The question is:
* How many sub libraries does it need to be split into? FP classification is one, constants another, possibly rounding functions one more, then one more for everything else. Grief.
I am not sure the Grief need to be so bad. Are you assuming all modules have to be maintained in separate Git repositories? That does not have to be the case. I will try to explain better, i.e. with other words, what I have tried to say in past discussion on this topic. I will use Boost.Math as an example, but this applies to any Boost Library, maybe in particular to Config which configure dependencies to a large number of technology dependent supporting features. Those features can be seen as modules as well of you follow my line of though. So, let us say Math repository is *not* split in any way but continue more or less as is, but is modularized somehow. What could that look like: First of all, everything that is supporting to the library itself such as tests, examples, docs, etc. are defined as separate modules. I think these already live in their separate sub-directories and have their separate build mechanics. None of this incur dependencies in the Math Library modules themselves. Then, everything in the library itself is split into modules based on some analysis of dependencies and re-engineering deemed needed to reduce those dependencies. The source and headers does not necessarily need to be moved around to make it explicit in the file structure of the repository. That could be at the discretion of library maintainers. However, the library maintainer need to support some means for external tools to identify the content of the modules and their direct dependencies. Such tools could then support various packaging and deployment needs. The granularity of modules may vary a-lot. Not accepting that will create problems. As example, there is some talk of optional headers in this discussion, in my mind that is simply a separate module. It is no need to require such single header modules to be moved into their own library, or to attempt to collect such headers with related dependencies into some common module. That is just turning the problem around, mot really solving anything. To understand where modularizing is going, and thus be able to answer the poll with less ambiguities. I think some fundamentals need to be agreed. Some issues in that regard is at least unclear to me, or I expect there to be other views than mine on the list if we where asked questions, like for example: - What is a module? - What is the relationship between modules and libraries? - What is the relationship between modules and source repositories? - When does a dependency between two modules occur? - What sort of dependencies do we want to prevent occurring? Intentions of those willing to proceed with work on modularizing boost is not clearly stated. References to past discussions are given, but no conclusions about how to proceed. I think that is intentional as some of the process may best be agreed to later. However there is a danger of starting off in the wrong direction. So maybe we should try to agree on some ground rules for modules in boost. Below are my current notions of how the above questions should be answered. I expect other to have other views. ----------- What is a module? ------------- A module can be a single file or a collection of files (e.g.: header and source files) that are hard or impossible to use without the rest of the module. A module is self contained, except that it may have defined dependencies to other modules incurring when it is used by a given tool-set. ---- What is the relationship between modules and libraries? ----- A library has a maintainer individual or group and is maintained in a separate repository from other libraries in Boost. A library may contain many modules. ---- What is the relationship between modules and repositories? ----- Repositories are used to maintain configuration control of a library or some other set of modules in boost. A repository may contain many modules. ---- When does a dependency between two modules occur? ----- The dependency occur when a certain tool processing the dependent module uses or is attempting to use the module it is depending upon. In the latter case we have a failure, which we clearly want to prevent, if that is not feasible, then failures should at least be simple to resolve by identifying the dependency and get it properly deployed onto the system. ---- What sort of dependencies do we want to prevent occurring? ---- Dependencies to modules providing features the user is not requiring. Dependencies to modules using technology the user is not utilizing. -- Bjørn
On Saturday 31 May 2014 13:23:56 Bjørn Roald wrote:
I am not sure the Grief need to be so bad. Are you assuming all modules have to be maintained in separate Git repositories? That does not have to be the case.
Can you actually download a part of a git repository? If not, I think this kind of defeats the modularization purpose, which is basically to reduce the amount of code you need to download to obtain a minimal part of Boost. E.g. to use Boost.Conversion you would still need to download the whole Boost.Math.
On 05/31/2014 01:41 PM, Andrey Semashev wrote:
On Saturday 31 May 2014 13:23:56 Bjørn Roald wrote:
I am not sure the Grief need to be so bad. Are you assuming all modules have to be maintained in separate Git repositories? That does not have to be the case.
Can you actually download a part of a git repository? If not, I think this kind of defeats the modularization purpose, which is basically to reduce the amount of code you need to download to obtain a minimal part of Boost. E.g. to use Boost.Conversion you would still need to download the whole Boost.Math.
That is assuming you need to use Git as the means to deploy boost Boost.Conversion onto your system. That may be a fair assumption if you are a boost developer, but it need not be the case if you are a user. Besides that I think Conversion already lives in its own Git repo. Further, my way of thinking about modules will not prevent maintainers of Boost.Math or other libraries to split or move modules into multiple repositories. However, forcing a one module in one repository policy will have a lot of issues, and IMHO it will not be feasible to make a clean model of what modules are. -- Bjørn
On 05/31/2014 01:41 PM, Andrey Semashev wrote:
On Saturday 31 May 2014 13:23:56 Bjørn Roald wrote:
I am not sure the Grief need to be so bad. Are you assuming all modules have to be maintained in separate Git repositories? That does not have to be the case.
Can you actually download a part of a git repository?
Not with Git in any straight forward way.
If not, I think this kind of defeats the modularization purpose, which is basically to reduce the amount of code you need to download to obtain a minimal part of Boost.
Defining that a repository can consist of multiple modules, does not prevent reducing dependencies between repositories. Dependencies only matter when you use some tool on a module. If that tool is a C++14 compiler which does not require some support module to be available to emulate a certain C++14 feature, then there is no dependency. If that support feature live in some other module, then that is not needed. You do not need to fetch the repository containing the module. Another example is if you have a serialization support module in a library/repo, maybe a single header file. That module would only incur a dependency to the serialization library/repo if you are in fact using serialization, in which case there is no problem. You do not need to fetch the serialization library if you do not use it. -- Bjørn
On 31 May 2014 12:23, Bjørn Roald <bjorn@4roald.org> wrote:
---- What is the relationship between modules and libraries? ----- A library has a maintainer individual or group and is maintained in a separate repository from other libraries in Boost. A library may contain many modules.
We already have multiple libraries in a repository. For example: https://github.com/boostorg/functional/blob/develop/meta/libraries.json These metadata files are only in a few libraries at the moment, but dependency specification could possibly be added to them. Although if it's going to be done for subsets of libraries, it would need to be in a separate file or files.
On 05/31/2014 04:22 PM, Daniel James wrote:
On 31 May 2014 12:23, Bjørn Roald <bjorn@4roald.org> wrote:
---- What is the relationship between modules and libraries? ----- A library has a maintainer individual or group and is maintained in a separate repository from other libraries in Boost. A library may contain many modules.
We already have multiple libraries in a repository.
I see no reason to disallow multiple libraries in a repository. If those libraries are related through a group of maintainers working well together and the libraries cover the same problem domain, then it may be more natural and simpler. Splitting boost into multiple repositories has to do with foreseen scaling issues for the overall boost effort. As long as those goals are not obstructed by larger repositories there should be no problem. Forcing such libraries or modules into their separate Git repository is not meaningful. On the contrary, enforcing such a policy will either lead to too many repositories which is a challenge for humans and technology, or too few modules, which will make it very hard to deal with dependencies.
For example:
https://github.com/boostorg/functional/blob/develop/meta/libraries.json
These metadata files are only in a few libraries at the moment, but dependency specification could possibly be added to them.
Interesting, I take it these are the modular version of the information needed to produce the listing of Boost Libraries for the web site and documentation. It certainly could be built upon.
Although if it's going to be done for subsets of libraries, it would need to be in a separate file or files.
Yes, or they could just be structure elements in the same meta file under each library entry. But I can see that may be undesirable if the module metadata become lengthy or complex. Could we think of it differently, that Library in the traditional Boost conventions are just a form of module that may have relationship to any number of other modules depending on it. Such modules may be part of the Library as far as authorship and maintenance if there is a ownership relation "owned-by", defined in the meta data of the dependent module. It should be considered natural, but maybe not mandatory, that a module is in the same repository as the module that it is owned-by. Thus e.g.: a test module can have separately defined dependencies from the library itself. What we call the structures boost is organized into need to be agreed and understood. From legacy, library is an obvious structure element. Whether the modules that we should be thinking of when we attempt to modularize boost are structured as libraries, units, components, package, files, directories, repositories or some combination of these is not clear. It make sense to have some constraints to what a module can be if metadata need to be recorded and tools need to support higher level structures. Whatever is decided, if something in addition to repository and library need to be given a name. As of now, I suspect that what I think of as a module in general terms, to others the same structure elements may be a sub-library, a library support directory such as doc, example, test, or a feature support file. They may not understand that I am thinking of all of these as modules. Common terminology is not important as long as we understand each other, but it makes understanding harder. To fix that, it may help with a common set of defined terms. -- Bjørn
John Maddock wrote:
On 30/05/2014 10:44, Andrey Semashev wrote:
On Thursday 29 May 2014 22:22:05 Julian Gonggrijp wrote:
PROPOSAL
The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:
1. Reduction of dependencies between Boost libraries.
This is a worthy goal, I don't think anyone reasonably disagrees with this.
Agree, to a reasonable point. I don't think that solid libraries should be torn apart or unrelated components of different libraries should be mixed together just to reduce dependencies. The cases where this would be beneficial should be discussed with library maintainers.
I think that notion of optional dependencies it also needed to achieve this goal. There are multiple places in Boost where dependencies are intentionally loosened but formally exist.
Right the devil is in the detail.
For example the Math lib is one of those with cyclic dependencies - there's a small core that a few libraries depend on, but which Math also uses to some degree (like lexical_cast).
The question is:
* How many sub libraries does it need to be split into? FP classification is one, constants another, possibly rounding functions one more, then one more for everything else. Grief. * How can such a split be done and yet still provide a seemless experience for the user - i.e. I'm strongly in favour of keeping the docs all in one place, and the header directory structure as it is now, while possibly allowing users that are just using say lexical_cast to not have to download the whole Math lib. * Ideally I'd like to keep maintenance overhead as low as possible - the change to the new Git directory structure has been quite disruptive, and everyone involved in maintaining the Math lib has got into "header hell" more than once by editing the wrong file. Splitting the lib makes that even worse.
In general, of course we should not cut dependencies at all costs. From what you say I get the impression Boost.Math might actually be best off as a monolithical submodule.
We also need sensible policies for dealing with optional components - a good example would be libraries that provide serialization support in a separate optional header. The library as such does not require Boost.Serialization, but quite rightly the optional "bridging" support is there. I asked about this last time this topic came up, but I saw no good answer?
I think what you describe here would be a good solution.
Please note: *I still think reducing dependencies is a good idea*, but I'd really like to see some kind of workflow that addresses these issues, otherwise I fear actually making things worse if libraries get harder to maintain at the alter of reduced dependencies.
So far there seems to be quite some support for the general idea of dependency reduction and automated dependency cloning, so I'm willing to write a more detailed proposal that includes (as unintrusive as possible) policies for library maintainers when I find the time. -Julian
Julian Gonggrijp wrote:
PROPOSAL
The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:
1. Reduction of dependencies between Boost libraries. 2. Simple but effective automation of dependency handling.
We all agree on these goals in principle, but what do you actually propose we need to do? We need to start somewhere. Specifics are needed. What I can contribute at the moment is - a suggestion that we publish a Boost dependency report in HTML format somewhere on boost.org that tells us, and anyone interested, what depends on what and why; - a program that generates such a report (in plain text) as a starting point (attached). As an example, I've pasted inline the output of "boostdep smart_ptr". This only gives the primary dependencies; it would be nice for the finished product to include the secondary module dependencies as well (without going into specifics what included what, just what module brought up the secondary dependency). Dependencies for smart_ptr: align: <boost/align/align.hpp> from <boost/smart_ptr/detail/array_allocator.hpp> assert: <boost/assert.hpp> from <boost/smart_ptr/detail/atomic_count_pt.hpp> from <boost/smart_ptr/detail/lwm_pthreads.hpp> from <boost/smart_ptr/detail/sp_counted_base_pt.hpp> from <boost/smart_ptr/detail/spinlock_nt.hpp> from <boost/smart_ptr/enable_shared_from_raw.hpp> from <boost/smart_ptr/enable_shared_from_this.hpp> from <boost/smart_ptr/intrusive_ptr.hpp> from <boost/smart_ptr/scoped_array.hpp> from <boost/smart_ptr/scoped_ptr.hpp> from <boost/smart_ptr/shared_array.hpp> from <boost/smart_ptr/shared_ptr.hpp> <boost/current_function.hpp> from <boost/detail/sp_typeinfo.hpp> config: <boost/config.hpp> from <boost/detail/lightweight_thread.hpp> from <boost/detail/sp_typeinfo.hpp> from <boost/get_pointer.hpp> from <boost/smart_ptr.hpp> from <boost/smart_ptr/detail/array_utility.hpp> from <boost/smart_ptr/detail/atomic_count.hpp> from <boost/smart_ptr/detail/lightweight_mutex.hpp> from <boost/smart_ptr/detail/quick_allocator.hpp> from <boost/smart_ptr/detail/shared_count.hpp> from <boost/smart_ptr/detail/sp_convertible.hpp> from <boost/smart_ptr/detail/sp_counted_base.hpp> from <boost/smart_ptr/detail/sp_counted_impl.hpp> from <boost/smart_ptr/detail/sp_forward.hpp> from <boost/smart_ptr/detail/sp_interlocked.hpp> from <boost/smart_ptr/detail/sp_nullptr_t.hpp> from <boost/smart_ptr/detail/spinlock.hpp> from <boost/smart_ptr/detail/spinlock_pool.hpp> from <boost/smart_ptr/detail/yield_k.hpp> from <boost/smart_ptr/enable_shared_from_raw.hpp> from <boost/smart_ptr/enable_shared_from_this.hpp> from <boost/smart_ptr/intrusive_ptr.hpp> from <boost/smart_ptr/intrusive_ref_counter.hpp> from <boost/smart_ptr/make_shared_object.hpp> from <boost/smart_ptr/make_unique_object.hpp> from <boost/smart_ptr/scoped_array.hpp> from <boost/smart_ptr/scoped_ptr.hpp> from <boost/smart_ptr/shared_array.hpp> from <boost/smart_ptr/shared_ptr.hpp> <boost/config/no_tr1/functional.hpp> from <boost/smart_ptr/intrusive_ptr.hpp> <boost/config/no_tr1/memory.hpp> from <boost/get_pointer.hpp> from <boost/smart_ptr/detail/shared_count.hpp> from <boost/smart_ptr/shared_ptr.hpp> <boost/detail/workaround.hpp> from <boost/smart_ptr/detail/shared_count.hpp> from <boost/smart_ptr/detail/sp_counted_base_w32.hpp> from <boost/smart_ptr/enable_shared_from_raw.hpp> from <boost/smart_ptr/intrusive_ptr.hpp> from <boost/smart_ptr/scoped_array.hpp> from <boost/smart_ptr/scoped_ptr.hpp> from <boost/smart_ptr/shared_array.hpp> from <boost/smart_ptr/shared_ptr.hpp> exception: <boost/throw_exception.hpp> from <boost/smart_ptr/detail/shared_count.hpp> from <boost/smart_ptr/shared_ptr.hpp> type_traits: <boost/type_traits/add_rvalue_reference.hpp> from <boost/smart_ptr/make_unique_object.hpp> <boost/type_traits/alignment_of.hpp> from <boost/smart_ptr/detail/array_allocator.hpp> from <boost/smart_ptr/detail/quick_allocator.hpp> from <boost/smart_ptr/make_shared_object.hpp> <boost/type_traits/has_trivial_constructor.hpp> from <boost/smart_ptr/detail/array_utility.hpp> <boost/type_traits/has_trivial_destructor.hpp> from <boost/smart_ptr/detail/array_utility.hpp> <boost/type_traits/remove_cv.hpp> from <boost/smart_ptr/detail/array_traits.hpp> <boost/type_traits/type_with_alignment.hpp> from <boost/smart_ptr/detail/quick_allocator.hpp> from <boost/smart_ptr/make_shared_object.hpp> utility: <boost/checked_delete.hpp> from <boost/smart_ptr/detail/shared_count.hpp> from <boost/smart_ptr/detail/sp_counted_impl.hpp> from <boost/smart_ptr/scoped_array.hpp> from <boost/smart_ptr/scoped_ptr.hpp> from <boost/smart_ptr/shared_array.hpp> from <boost/smart_ptr/shared_ptr.hpp> <boost/utility/addressof.hpp> from <boost/smart_ptr/detail/shared_count.hpp>
This only gives the primary dependencies; it would be nice for the finished product to include the secondary module dependencies as well (without going into specifics what included what, just what module brought up the secondary dependency).
And here's the secondary report on smart_ptr. (My initial version listed modules in alphabetical order, not in discovery order, which was even more stunning as I hadn't the faintest idea why all these modules were deemed a dependency.) Secondary dependencies for smart_ptr: align: adds integer adds static_assert exception: adds smart_ptr adds tuple type_traits: adds mpl adds preprocessor adds static_assert adds typeof utility: adds iterator adds mpl adds preprocessor adds static_assert iterator: adds concept_check adds conversion adds detail adds function_types adds optional mpl: adds detail concept_check: adds parameter conversion: adds array adds container adds math adds numeric/conversion adds range detail: adds predef optional: adds move array: adds functional container: adds functional adds intrusive math: adds algorithm adds atomic adds format adds function adds fusion adds multiprecision adds numeric/ublas adds random adds regex adds test adds tr1 parameter: adds python range: adds algorithm adds functional adds regex algorithm: adds bind function: adds bind fusion: adds *unknown* multiprecision: adds rational numeric/ublas: adds *unknown* adds serialization python: adds bind adds foreach random: adds system test: adds bind adds io adds timer tr1: adds bind adds unordered serialization: adds spirit adds variant timer: adds chrono chrono: adds accumulators adds interprocess adds ratio spirit: adds iostreams adds phoenix adds pool adds proto adds thread accumulators: adds circular_buffer interprocess: adds date_time thread: adds date_time date_time: adds tokenizer
On Fri, May 30, 2014 at 5:02 PM, Peter Dimov <lists@pdimov.com> wrote:
This only gives the primary dependencies; it would be nice for the
finished product to include the secondary module dependencies as well (without going into specifics what included what, just what module brought up the secondary dependency).
And here's the secondary report on smart_ptr. (My initial version listed modules in alphabetical order, not in discovery order, which was even more stunning as I hadn't the faintest idea why all these modules were deemed a dependency.)
Secondary dependencies for smart_ptr:
<snip>
range: adds algorithm adds functional adds regex
</snip> I'm not sure this report is giving an accurate representation of the true dependencies of smart_ptr. Boost.Range only depends upon Boost.Algorithm in the unit tests and within the collection_traits which I can't imagine you would use for a smart_ptr. Similarly if you are not using the regex functionality, smart_ptr will not be dependent on it. The collection_traits and regex includes are very easily avoided and I believe smart_ptr does not really depend on these at all. The dependency on Boost.Functional only occurs if you explicitly include boost/range/iterator_range_hash.hpp I think that the conditional dependencies by including subsets of headers is quite frequently used and therefore analysis needs to take this into account. It could very well be argued that the RegEx adaptor would be better implemented in the RegEx library. It could also be argued that the Range-based algorithms might be better placed in Boost.Algorithm. My rationale for them being in Boost.Range is merely that I have write access and permission to develop Boost.Range. The coordination effort between libraries would have been prohibitive initially. If I am mistaken about your dependencies then please let me know as I certainly deem it unacceptable that you should not be able to avoid dependencies on Boost.Regex in particular. I do however agree with one of your earlier comments to the effect that modularisation would not improve external quality factors for users but would require a large development cost. I'm sure we can achieve a lot more useful things with the same development effort. Regards, Neil Groves
On Fri, May 30, 2014 at 9:02 AM, Peter Dimov <lists@pdimov.com> wrote:
exception: adds smart_ptr adds tuple
This was the most surprising to me. I assumed, and now really desire, exception to not have any other dependencies.
utility: [snip: Long list]
This is the big offender which results in the whole world being pulled in (exaggeration for effect), and we both use it only for boost::addressof. Glen
Glen Fernandes wrote:
On Fri, May 30, 2014 at 9:02 AM, Peter Dimov <lists@pdimov.com> wrote:
exception: adds smart_ptr adds tuple
This was the most surprising to me. I assumed, and now really desire, exception to not have any other dependencies.
It's a false dependency. 'exception' only depends on 'tuple' if you include one specific header. (I think that Emil was considering its removal.) In addition, we only use throw_exception.hpp, which actually doesn't even need 'exception'; it's (almost) self-contained. It could be extracted into its own 'throw_exception' module. Neil Groves wrote:
I'm not sure this report is giving an accurate representation of the true dependencies of smart_ptr.
It contains a lot of false dependencies that are brought in because module X needs Y1.hpp in module Y, which also contains Y2.hpp, which needs Z, which is not needed by X. But that is just how module-level dependencies work. If we adopt the simple scheme of specifying dependencies on a module basis, X will list Y and Y will list Z, and we'll get the result I posted. I could have built the transitive closure of the actual #includes, and it would have given a different result, but that's not what I'm emulating.
This only gives the primary dependencies; it would be nice for the finished product to include the secondary module dependencies as well (without going into specifics what included what, just what module brought up the secondary dependency).
And here's the secondary report on smart_ptr. (My initial version listed modules in alphabetical order, not in discovery order, which was even more stunning as I hadn't the faintest idea why all these modules were deemed a dependency.)
A lot of those look like testing dependencies rather than header-usage dependencies? John.
John Maddock wrote:
A lot of those look like testing dependencies rather than header-usage dependencies?
No, they are purely header-based. They don't even take the library sources into account. Only headers are scanned. If you want to see why module X is listed as requiring module Y, compile boostdep.cpp, cd into the Boost root and run "boostdep X" and it will tell you.
Peter Dimov wrote:
Julian Gonggrijp wrote:
PROPOSAL
The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:
1. Reduction of dependencies between Boost libraries. 2. Simple but effective automation of dependency handling.
We all agree on these goals in principle, but what do you actually propose we need to do? We need to start somewhere. Specifics are needed.
By now it is clear to me that more specifics are needed. I'm looking out for time to work on that.
What I can contribute at the moment is
- a suggestion that we publish a Boost dependency report in HTML format somewhere on boost.org that tells us, and anyone interested, what depends on what and why;
- a program that generates such a report (in plain text) as a starting point (attached).
Thank you for this contribution, Peter! Ideally the HTML report is generated from the latter plaintext, of course. In order for your program to provide input for the automated downloader, I think it would have to provide information at a coarser level of detail, i.e. entire submodules. This is something we can figure out in due time. -Julian
Julian Gonggrijp wrote:
Thank you for this contribution, Peter! Ideally the HTML report is generated from the latter plaintext, of course.
The next version of the program I posted has the option of generating HTML directly. You've probably seen the output. If not, do look at a few modules; it's good to have an idea of what our future automated downloader would do when you tell it to download, say, smart_ptr (in the present state of Boost).
In order for your program to provide input for the automated downloader, I think it would have to provide information at a coarser level of detail, i.e. entire submodules.
But it does provide the information on a submodule level. It just goes into more detail to explain the dependency, but this additional output can be trivially suppressed. :-)
Peter Dimov wrote:
Julian Gonggrijp wrote:
Thank you for this contribution, Peter! Ideally the HTML report is generated from the latter plaintext, of course.
The next version of the program I posted has the option of generating HTML directly. You've probably seen the output. If not, do look at a few modules; it's good to have an idea of what our future automated downloader would do when you tell it to download, say, smart_ptr (in the present state of Boost).
Great, I will have a look at it when I'm back in a more programmer-friendly environment. That will take a few more days, please bear with me in the meanwhile. ;-)
In order for your program to provide input for the automated downloader, I think it would have to provide information at a coarser level of detail, i.e. entire submodules.
But it does provide the information on a submodule level. It just goes into more detail to explain the dependency, but this additional output can be trivially suppressed. :-)
Excellent!
On Thu, May 29, 2014 at 3:22 PM, Julian Gonggrijp <j.gonggrijp@gmail.com> wrote:
Dear Boost mailing list members,
I would like to revive the topic of Boost inter-library dependencies, suggest two global changes to Boost and ask whether you agree with the course of action I propose.
The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:
1. Reduction of dependencies between Boost libraries. 2. Simple but effective automation of dependency handling.
I am in favor. With #2, as others have mentioned, it should be enumerated by dependency for each supported toolset. (which also gives a nice list of supported toolsets for that library) Tom Kent
Dear thread participants, I am currently unable to access the list (or anything internet-related whatsoever), my apologies for lack of response on my part so far. I hope to be able to respond soon again. Please keep voting and thanks to everyone who responded so far. Cheers, Julian
Julian Gonggrijp wrote:
[...]
PROPOSAL
The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:
1. Reduction of dependencies between Boost libraries. 2. Simple but effective automation of dependency handling.
POLL
I think it may be easier for the steering committee to justify any decision on this proposal, if they have hard statistics about the amount of support it receives. Therefore, I invite every opinionated person reading this to reply with a vote, either in favour of the proposal or against it. I invite you to do this even if you do not feel the need or find the time to provide arguments for your position. You can later re-vote if you change your mind.
I will count the votes, and when there are more than 30 I will start posting the statistics occasionally. Votes are public so everyone can verify my data when necessary. Anyone, including the steering committee, can then use the numbers however they choose.
Much less than 30 (explicit) votes have come in but this thread seems to have come to an end, so it is time for a closing report. There is strong and unanimous support for part 1 of the proposal. I am delighted that several people are currently fanatically working on dependency reduction. :-) Part 2 also found strong support, although it faced more reservations. This is in part due to the lack of a draft implementation. I plan to write that draft as soon as I find the time (this may take in the order of months). My current idea about the automated dependency handling tool is as follows. - The dependency tool will use or adapt Peter Dimov's boostdep tool to list direct dependencies for a single module at a time. - Each module will contain a single, small, static, versioned configuration file which lists the direct dependencies in the same way that the tool does. - Dependencies listed in the configuration file may have annotations to mark dependencies that are tool-specific or optional. For the time being, these annotations will not be added automatically. - The configuration file may initially be generated using the dependency tool (without annotations), or written by hand. - The configuration file may be updated both by hand and by running the dependency tool, which extends the file with any new dependencies that may have come up and prunes any dependencies that have been removed. Pre-existing annotations will be preserved (unless a dependency is pruned). - The dependency tool will run as part of the test suite to verify that the configuration file is complete. Of course this check can also be run separately. - The dependency tool will run as part of the bootstrap script and/or Boost.Build to automatically and recursively clone dependencies when needed. This can also be done separately. - Verification and cloning will be tool-aware. - Optional dependencies are ignored, they can still be cloned by explicitly requesting them as targets. - Handling of tool-specific and optional dependencies, as well as the corresponding annotations in configuration files, will be implemented after the other features. Feedback is welcome. Cheers, Julian
participants (15)
-
Andrey Semashev -
Bjorn Reese -
Bjørn Roald -
Daniel James -
Glen Fernandes -
J Gonggrijp -
John Maddock -
Julian Gonggrijp
-
Nat Goodspeed -
Neil Groves -
Niall Douglas -
Peter Dimov -
Stefan Seefeld -
Stephen Kelly -
Tom Kent