[tools][bcp] using boost internally in other libraries
Hi boosters, In the following I have made the assumption that it is in the spirit of the Boost license for Boost to be used as foundation for other libraries, open-source, free, commercial, or whatever. In addition, that any modifications to the libraries is allowed for any such purpose. However, it is my belief that most of the developers would not like to see extensive branching in the effort of providing better Boost libraries. This would most likely stretch the potential of the C++ community in providing a better future set of standard libraries for the language. So to my main subject: I have been working for some time to get acceptance for using Boost libraries as elements in a higher level portable platform for library development. It has surprised me how hard it is to make compelling and attractive set of arguments for using Boost (or other libraries) without attempting to ignore some pretty serious concerns. These concerns are: - commercial support for our use of Boost - quality assurance of Boost itself - stability of Boost API - maintainability of Boost source code - deployability of our library into environments using different versions of Boost or libraries used in boost - configuration management issues in general for us and our customers - platform support in Boost ( or if the platform support Boost if you like ;-) All of these concerns can be declared acceptable, given that you are willing to accept trade offs and risk. The issue I try to raze is if it is possible to reduce the magnitude of these risks and trade offs. One of the key issues is that you have to decide on some strategy of how to maintain your boost version. The most obvious strategy may be to stay with the latest Boost release at any time, this will however add risk as you have to adopt new changes as they are introduced into Boost and not as it fits your own release plans. The obvious alternative is to stay with a Boost release longer, possibly until you are forced to switch. This strategy will quickly force you into maintaining your own branch as Boost developers can not be expected to support old releases. Between these there are many shades of gray which may fit more or less well depending on your project and your customers. If your customers themselves use boost you are likely to meet a added configuration management, testing and release related cost and risk. As library developers have a very different type of customer relationship than what is the case for end application developers there are a number of concerns that make adaptation of Boost, or any other 3rd party library for that sake, more problematic for library developers. The main issue is to what degree a library depends on other libraries and hence introduces indirect or direct dependencies to its users. These dependencies tend to be troublesome for the library developer to support, as well as they may be troublesome for users to accept. In fact users may easily be in situations where such dependencies are completely unacceptable. To help to understand the issue I like to define categories of libraries which affect the degree they are affected by these concerns. Cat1. open source, source distributions, allowing encapsulation Cat2. open source, source distributions in general Cat3. open source, binary distributions Cat4. closed source Cat1 and Cat2 are build-able source code distributions. This give the user the ability to build the library and ensure that it is built correctly with other library it depends on, on the target system. Cat1 add to this the ability to hide the use of it inside other libraries, i.e. it can live concurrently in an end system with other versions of the same Cat1 library. Cat3 and Cat4 are for this discussion as far as I can tell similar. Cat3 may be called open source by some, but if for a variation of reasons rebuild by user is not an option, the results in this discussion becomes the same. Boost is a Cat2 library. Even if the extensive use of headers only libraries help some of the issues addressed in this post, it is not a general solution that makes Boost or part of Boost a Cat1 library. Now an example showing some dependencies: MyCustomersApplication |---- Boost 1.33 |---- MyLib |---- Boost 1.32 If MyLib is Cat1 or Cat2 the chances are good that we may be ok. Most likely MyLib will soon adopt the migration to new Boost releases by the actions of the maintainer or by patches from end users. If however MyLib is Cat 3 or 4 we have a problem. This problem is real even if the use of Boost in MyLib is strictly internal. In my view this is an especially bad problem. Even if I do not intend to use Boost in any way in the API of MyLib I still have this issue of incompatibility. This problem will disqualify use of Boost even internally for many library developers. Now, if we assume (which in most cases is dangerous) that library developers are a skilled race among the general population of the C++ community, it is in my view sad that many of them are not able to use Boost or potentially participate professionally in the efforts to improve Boost. I am sure many actually would participate if they could use it at work. Any management with commercial goals mixed with good sense of business would also let their developers participate actively, especially if this was the best way of maintaining parts of their product. This would let their developers focus their effort on the parts of the product that is core to their business rather than the parts that can be found and downloaded on the net (read boost.org). Ok, back to the example. If MyLib have no intention of using Boost in the API, is there a way? I have been struggling with this issue for a while. The obvious solution to me is to make Boost a Cat1, the problem is how? As mentioned earlier in many threads on this list header-only libraries helps, however this does not apply to all of Boost and it is possibly other issues that I am not aware of. If we look at the C++ domain in general, there are limited ways of hiding symbols in libraries. As far as I can see, none of these are sufficiently powerful to hide boost completely. One alternative is however to let the symbols leak, but make sure none are in conflict with the other version of boost. Some sort of namespace replacement or wrapping comes to my mind. Side Note: I am aware of symbol encapsulation abilities in windows DLLs and similar new features supported by GCC, however these are not generally supported on all platforms, so I have selected to ignore them. They may however be helpful to many. The general idea is that there are no references to symbols in the boost namespace between any MyLib libraries, nor in any header files in or referenced from MyLib API. How can this be achieved? I have concluded that accessing boost trough namespace wrapping header files are unpractical at best. This needs to be designed into all of boost for this to work, and I do not know exactly how. The main challenge is #include <boost/xxxx.hpp> directives which does not refer to the wrapping headers. An alternative is to replace all references to boost namespace with e.g. MyLib::boost namespace in the boost source code and change all #include <boost/xxx.hpp> directives to #include <Mylib/boost/xxx.hpp>. This would make a buildable library which would not share symbols with other versions of boost. The result is a branch or clone if you like of Boost that can be used with the same effect as if we actually had a Cat1 boost library. There is however serious problems here. Any management with common sense would say.... hey.... stop. The prospect of undertaking maintenance of a branch or clone of Boost does not make much sense --- So, the only way this would work is to make it possible to maintain the library in the boost community and somehow make the translation to the MyLib/boost namespace a pure production step. In configuration management terms this means that the original or possibly slightly patched boost source code is under configuration control along with a tool-chain used under MyLib system build that transforms the boost source tree to the form used internally. This is where I started looking at Boost tools/bcp. The bcp (boost copy I guess) tool is an excellent starting point since it already has the ability to copy selected libraries and their dependencies from the boost distribution to project specific source tree. So in a sense it is similar in spirit, but solves primarily a different goal, which is to reduce the size of the boost distribution imported. This is complementary to what I want. I have experimented a little with some extensions to bcp using some simple regexp replace calls and got some encouraging results. I am able to build the resulting source tree using only small adjustments to top level bjam files after treatment by the modified bcp. I have yet to complete support for replacing boost with nested namespaces since it is trickier to do the replacement of the ending braces of the namespace scope. I believe other tools than the Regexp library may be more appropriate for this task. So to wrap up: I do not believe anything prevents me from going along this path if I decide to do so. I am however interested in the boost developers attitude toward this approach, and in particular if this is something you will encourage and/or support. I am willing to work on the tool support for this in boost or to maintain a modified bcp tool if that is more appropriate. Regards Bjørn Roald
Bjørn, A very interesting read, thankyou for raising this.
- commercial support for our use of Boost
Just out of interest what kind of support would be most beneficial to commercial users?
- quality assurance of Boost itself
We're working on improving our release stategy, but any ideas on how we can do better would always be appreciated.
- stability of Boost API
That one is tricky, Boost is meant to be at least a little experimental.
- maintainability of Boost source code
That's a genuine concern, we have some code that can probably only be maintained by the folks who wrote it, and that's not good at all.
referenced from MyLib API. How can this be achieved? I have concluded
The general idea is that there are no references to symbols in the boost namespace between any MyLib libraries, nor in any header files in or that accessing boost trough namespace wrapping header files are
unpractical at best. This needs to be designed into all of boost for this to work, and I do not know exactly how. The main challenge is #include <boost/xxxx.hpp> directives which does not refer to the wrapping headers.
I think there are two issues here: 1) Side by side binaries: requires that symbol names in binaries are unique to the Boost version being used, or even the library being compiled. Requires that the Boost namespace be customisable to something other than "boost". I guess we could do what STLport does and make the namespace a macro throughout Boost, or we could do as you suggested and do a search and replace on "boost", unfortunately the latter is likely to be error prone at best. 2) Side by side headers: only needed if the library exposes it's Boost dependency in it's public API's. Requires that the #include's get fixed up to include via a path other than boost/. This is not as easy as it sounds, because there are some #include's in Boost that proceed via macros. Macro-ising this in the Boost source is probably not going to be easy or acceptible, but I'd be interested in what other folks feel. One benefit of placing all this inside the Boost source directly is that we could regression test it just by running some of our tests with BOOST_NAMESPACE (or whatever it's called) defined to something other than boost.
<boost/xxx.hpp> directives to #include <Mylib/boost/xxx.hpp>. This would make a buildable library which would not share symbols with other >versions of boost. The result is a branch or clone if you like of Boost that can be used with the same effect as if we actually had a Cat1 boost >library. There is however serious problems here. Any management with common sense would say.... hey.... stop. The prospect of undertaking >maintenance of a branch or clone of Boost does not make much sense --- So, the only way this would work is to make it possible to maintain the >library in the boost community and somehow make the translation to the MyLib/boost namespace a
An alternative is to replace all references to boost namespace with e.g. MyLib::boost namespace in the boost source code and change all #include pure production step. In configuration >management terms this means that the original or possibly slightly patched boost source code is under configuration control along with a tool-chain >used under MyLib system build that transforms the boost source tree to the form used internally. This is where I started looking at Boost >tools/bcp.
It would probably be easier (and more reliable actually) to change "boost" to "mylib_boost_1_33_0" or whatever rather than using nested namespaces, would that be an acceptable alternative? John.
John Maddock said:
Bjørn,
A very interesting read, thankyou for raising this.
thank you.
- commercial support for our use of Boost
Just out of interest what kind of support would be most beneficial to commercial users?
Hard to answer in a few words, but my personal feeling is that this is much less of a concern for developers than it is for managers. Many managers find it hard to trust something when they have nobody they can hit if things are not working out as expected. Personally I have experienced strong skepticism for ACE/TAO go away when a legal entity stated that they sell and support the tools commercially. You can also then assess this company and its solidity, which is a game they know how to play.
- quality assurance of Boost itself
We're working on improving our release stategy, but any ideas on how we can do better would always be appreciated.
I will keep that in mind, I know you are doing a lot and that is highly valued. Also if I may ad, your approaches to testing has been a good source of inspiration and solutions to me and my co-workers. The challenge may not be as much to do better, as there always is room for, but maybe to make potential users better aware of what is done.
- stability of Boost API
That one is tricky, Boost is meant to be at least a little experimental.
I know -- in a post some time back I mentioned that a sort of index of how stable the maintainer and boost developers considered the various accepted library APIs to be might be helpfull. I would have an easier time getting accept for using a library considered more stable by the developers. This information is not available to me now.
- maintainability of Boost source code
That's a genuine concern, we have some code that can probably only be maintained by the folks who wrote it, and that's not good at all.
This is closely linked with commercial support. If a product is commercially supported by somebody I trust, then I might not care. But if it is not, I will have to consider the possibility of having to fix problems myself. If I have no reasonable way of understanding the code, it spells TROUBLE.
referenced from MyLib API. How can this be achieved? I have concluded
unpractical at best. This needs to be designed into all of boost for
<boost/xxxx.hpp> directives which does not refer to the wrapping
The general idea is that there are no references to symbols in the boost namespace between any MyLib libraries, nor in any header files in or that accessing boost trough namespace wrapping header files are this to work, and I do not know exactly how. The main challenge is #include headers.
I think there are two issues here:
1) Side by side binaries: requires that symbol names in binaries are unique to the Boost version being used, or even the library being compiled. Requires that the Boost namespace be customisable to something other than "boost". I guess we could do what STLport does and make the namespace a macro throughout Boost,
This would give one source, better than mingling with source code. However I am not aware if a macro is a good solution for this, does that work well for STLport?
or we could do as you suggested and do a search and replace on "boost", unfortunately the latter is likely to be error prone at best.
agree that there are issues with a search and replace strategy, however it may be feasible.
2) Side by side headers: only needed if the library exposes it's Boost dependency in it's public API's. Requires that the #include's get fixed up to include via a path other than boost/. This is not as easy as it sounds, because there are some #include's in Boost that proceed via macros.
I do not understand exactly what you mean with "proceed via macros". Could you point me at an example please?
Macro-ising this in the Boost source is probably not going to be easy or acceptible, but I'd be interested in what other folks feel.
One benefit of placing all this inside the Boost source directly is that we could regression test it just by running some of our tests with BOOST_NAMESPACE (or whatever it's called) defined to something other than boost.
maybe even test a library's integration in in the "std" namespace, there will be some name conflict issues with the various standard libraries, but it feels like a tempting thing to try. Try the feel of your proposed library in the std namespace... demonstrate integration with std library implementations -- who knows.
<boost/xxx.hpp> directives to #include <Mylib/boost/xxx.hpp>. This would make a buildable library which would not share symbols with other versions of boost. The result is a branch or clone if you like of Boost that can be used with the same effect as if we actually had a Cat1 boost >library. There is however serious problems here. Any management with common sense would say.... hey.... stop. The prospect of undertaking >maintenance of a branch or clone of Boost does not make much sense --- So, the only way
An alternative is to replace all references to boost namespace with e.g. MyLib::boost namespace in the boost source code and change all #include this would work is to make it possible to maintain the >library in the boost community and somehow make the translation to the MyLib/boost namespace a pure production step. In configuration >management terms this means that the original or possibly slightly patched boost source code is under configuration control along with a tool-chain >used under MyLib system build that transforms the boost source tree to the form used internally. This is where I started looking at Boost >tools/bcp.
It would probably be easier (and more reliable actually) to change "boost" to "mylib_boost_1_33_0" or whatever rather than using nested namespaces, would that be an acceptable alternative?
Yes, I think I would just use "mylib_boost" since the version would be implicit through my configuration management. Hopefully my customers are only using one version of MyLib ;-) . But thats really their problem, not mine. Nesting would however be nice since we internally could write #using namespace MyLib; and then write boost code as everybody else. -- Bjørn
I know -- in a post some time back I mentioned that a sort of index of how stable the maintainer and boost developers considered the various accepted library APIs to be might be helpfull. I would have an easier time getting accept for using a library considered more stable by the developers. This information is not available to me now.
Good point.
- maintainability of Boost source code
That's a genuine concern, we have some code that can probably only be maintained by the folks who wrote it, and that's not good at all.
This is closely linked with commercial support. If a product is commercially supported by somebody I trust, then I might not care. But if it is not, I will have to consider the possibility of having to fix problems myself. If I have no reasonable way of understanding the code, it spells TROUBLE.
Understood.
1) Side by side binaries: requires that symbol names in binaries are unique to the Boost version being used, or even the library being compiled. Requires that the Boost namespace be customisable to something other than "boost". I guess we could do what STLport does and make the namespace a macro throughout Boost,
This would give one source, better than mingling with source code. However I am not aware if a macro is a good solution for this, does that work well for STLport?
Well I must admit I've never tried to change STLports namespace away from the default, but the intention is clearly there. In theory it's just a question of setting a #define somewhere and away you go.
2) Side by side headers: only needed if the library exposes it's Boost dependency in it's public API's. Requires that the #include's get fixed up to include via a path other than boost/. This is not as easy as it sounds, because there are some #include's in Boost that proceed via macros.
I do not understand exactly what you mean with "proceed via macros". Could you point me at an example please?
Sorry, I mean things like: #define MYHEADER <boost/whatever.hpp> // then much later, different source file maybe: #include MYHEADER Boost.Config does things like that, as does MPL I believe. However thinking about some more a global search and replace for "boost/" would probably be good enough.
Yes, I think I would just use "mylib_boost" since the version would be implicit through my configuration management. Hopefully my customers are only using one version of MyLib ;-) . But thats really their problem, not mine.
Nesting would however be nice since we internally could write
#using namespace MyLib;
and then write boost code as everybody else.
How about a namespace alias: namespace acme_computing_boost_1_33_0 {} namespace boost = acme_computing_boost_1_33_0; Is that an acceptable alternative? It could be boilerplated inside the Boost.Config system if necessary. John.
On Friday 23 September 2005 16:54, John Maddock wrote:
1) Side by side binaries: requires that symbol names in binaries are unique to the Boost version being used, or even the library being compiled. Requires that the Boost namespace be customisable to something other than "boost". I guess we could do what STLport does and make the namespace a macro throughout Boost,
This would give one source, better than mingling with source code. However I am not aware if a macro is a good solution for this, does that work well for STLport?
Well I must admit I've never tried to change STLports namespace away from the default, but the intention is clearly there. In theory it's just a question of setting a #define somewhere and away you go.
Yes it seams to be a reasonable assumption that it would work fine. However, I would not like to be the one that have introduced the idea of using #define BOOST_NAMESPACE boost simply because namespace boost { is much more pleasing to the eye than namespace BOOST_NAMESPACE { // ushhhjjj It does however open up for some interresting possibilities, by the way I think you came up with it ;-) kidding aside, see what I write below about your comments on side-by-side header support.
2) Side by side headers: only needed if the library exposes it's Boost dependency in it's public API's. Requires that the #include's get fixed up to include via a path other than boost/.
in addition, a macro based solution will not work well in a side-by-side header setting. The macros would expand to the same string when used in application. Hence different definitions of types would be used internally in the library from those in the API in context of the application. My main concern in my original post was support for side-by-side binaries. However if a relatively drastic solution is needed, we should consider side-by-side header support as well. To me it does not look like the macro solution support that.
This is not as easy as it sounds, because there are some #include's in Boost that proceed via macros.
I do not understand exactly what you mean with "proceed via macros". Could you point me at an example please?
Sorry, I mean things like:
#define MYHEADER <boost/whatever.hpp>
// then much later, different source file maybe:
#include MYHEADER
Boost.Config does things like that, as does MPL I believe. However thinking about some more a global search and replace for "boost/" would probably be good enough.
Ok, I see. I think as you point out that this would work fine. In my experimental extention to bcp I add a file header informing that the file is an autogenerated derivat of the original Boost file. Then I replace text based on a replacement namespace given as parameter. e.g. --namespace=MyLib::boost where the idea is to replace :: with / for paths and _ in macros. boost.org --> boost.org boost/ --> MyLib/boost/ boost --> Mylib::boost ( not implemented for nested ns yet ) BOOST --> MYLIB_BOOST This actually seemed to work surprisingly well. I have not done extencive testing on resulting code except for trying to compile some resulting mylib_boost libraries. That worked fine. There are possible traps, and the solution does not feel robust at all. I belive this solution need to be supported by a tool with much better inherent capabilities for C++ syntax than plain regexp replace, so what I have is nore like a prototype. I would not like boost developers to have to be careful not to break features in such a tool.
Yes, I think I would just use "mylib_boost" since the version would be implicit through my configuration management. Hopefully my customers are only using one version of MyLib ;-) . But thats really their problem, not mine.
Nesting would however be nice since we internally could write
#using namespace MyLib;
and then write boost code as everybody else.
How about a namespace alias:
namespace acme_computing_boost_1_33_0 {}
namespace boost = acme_computing_boost_1_33_0;
Is that an acceptable alternative? It could be boilerplated inside the Boost.Config system if necessary.
For side-by-side binaries that is fine, but not for side-by-side headers. But again, I could live with that. Side-by-side headers would however be nice to many, I think. If they really had different symbols in headers while compiling code using MyLib boost types could safely be used in API. How to deal with possible need for type convertion for types used in API, e.g. between Mylib::boost::filesystem::path and boost::filesystem::path, would then be a problem that the user of MyLib could handle in a variety of ways. An interresting thing I come to realize now is that my choice to look at a search replace strategy partly come from an interrest in not excluding posibility of using boost types in API. I think my preference would be not to use them to avoid symbol incompatibillities. But if they are in a different namespace they are no longer the boost types, or are they? ------ Bjørn
in addition, a macro based solution will not work well in a side-by-side header setting. The macros would expand to the same string when used in application. Hence different definitions of types would be used internally in the library from those in the API in context of the application.
My main concern in my original post was support for side-by-side binaries. However if a relatively drastic solution is needed, we should consider side-by-side header support as well. To me it does not look like the macro solution support that.
Correct.
Is that an acceptable alternative? It could be boilerplated inside the Boost.Config system if necessary.
For side-by-side binaries that is fine, but not for side-by-side headers. But again, I could live with that. Side-by-side headers would however be nice to many, I think. If they really had different symbols in headers while compiling code using MyLib boost types could safely be used in API. How to deal with possible need for type convertion for types used in API, e.g. between Mylib::boost::filesystem::path and boost::filesystem::path, would then be a problem that the user of MyLib could handle in a variety of ways.
Understood.
An interresting thing I come to realize now is that my choice to look at a search replace strategy partly come from an interrest in not excluding posibility of using boost types in API. I think my preference would be not to use them to avoid symbol incompatibillities. But if they are in a different namespace they are no longer the boost types, or are they?
Correct. There is one important use case for side by side headers though: where you want a private member of the class to be a boost type. An obvious example would be using a boost smart pointer to implement the PIMPL idiom. If a type is to be part of the API then the choices are: 1) Roll your own type with your own interface. 2) Roll your own implementation using the Boost (or TR1/2 interface). 3) Use the Boost implementation in your own namespace. 4) Use the Boost implementation as is. Option (2) leverages the familiarity that using an established interface brings, option (3) is effectively the same thing, but may introduce an ambiguity about who is responsible for the maintenance of that component, on the other hand Boost components are very widely used and tested, so may be a safer bet than a "new" implementation. Option (4) may be the ideal, but only if your library is Open Source (in the broadest possible meaning: in this case as long as the user can recompile with a Boost version of their choice). Just thinking out loud really... John.
Bjørn Roald <bjorn@4roald.org> writes:
I have been working for some time to get acceptance for using Boost libraries as elements in a higher level portable platform for library development. It has surprised me how hard it is to make compelling and attractive set of arguments for using Boost (or other libraries) without attempting to ignore some pretty serious concerns. These concerns are:
- commercial support for our use of Boost
I know of a certain company offering commercial support for boost :)
- quality assurance of Boost itself
Specifically?
- stability of Boost API
Yeah, there are no rules about it. However, we generally avoid breaking source compatibility.
- maintainability of Boost source code
I actually find that most of the time it's pretty easy to dig into a Boost library and fix things, although I seldom have to because the authors tend to be responsive to bug reports. However, there are no uniform standards for certain things that make maintenance easier for some people, like code formatting. But you probably had something else in mind?
- deployability of our library into environments using different versions of Boost or libraries used in boost
Hmm, that's a tough one, especially if your library traffics in boost types in its public API, or if you use compilers that don't support symbol visibility control.
- configuration management issues in general for us and our customers
Such as?
- platform support in Boost ( or if the platform support Boost if you like ;-)
Sorry; what do you mean?
All of these concerns can be declared acceptable, given that you are willing to accept trade offs and risk. The issue I try to raze is if it is possible to reduce the magnitude of these risks and trade offs.
Good question.
One of the key issues is that you have to decide on some strategy of how to maintain your boost version. The most obvious strategy may be to stay with the latest Boost release at any time, this will however add risk as you have to adopt new changes as they are introduced into Boost and not as it fits your own release plans. The obvious alternative is to stay with a Boost release longer, possibly until you are forced to switch. This strategy will quickly force you into maintaining your own branch as Boost developers can not be expected to support old releases.
A company offering commercial Boost support can support old releases by integrating fixes into older release branches. Whether or not those branches can ever become "official Boost releases" is not clear, but even if they don't, at least you can share a common set of fixes with everyone else who is depending on that old release.
Between these there are many shades of gray which may fit more or less well depending on your project and your customers. If your customers themselves use boost you are likely to meet a added configuration management, testing and release related cost and risk.
As library developers have a very different type of customer relationship than what is the case for end application developers there are a number of concerns that make adaptation of Boost, or any other 3rd party library for that sake, more problematic for library developers. The main issue is to what degree a library depends on other libraries and hence introduces indirect or direct dependencies to its users. These dependencies tend to be troublesome for the library developer to support, as well as they may be troublesome for users to accept. In fact users may easily be in situations where such dependencies are completely unacceptable. To help to understand the issue I like to define categories of libraries which affect the degree they are affected by these concerns.
Cat1. open source, source distributions, allowing encapsulation
Cat2. open source, source distributions in general
Cat3. open source, binary distributions
Cat4. closed source
Cat1 and Cat2 are build-able source code distributions.
I believe there are closed source libraries that are also delivered int the form of buildable source... but that's probably beside the point.
This give the user the ability to build the library and ensure that it is built correctly with other library it depends on, on the target system. Cat1 add to this the ability to hide the use of it inside other libraries, i.e. it can live concurrently in an end system with other versions of the same Cat1 library.
E.g. via symbol isolation or namespace renaming?
Boost is a Cat2 library. Even if the extensive use of headers only libraries help some of the issues addressed in this post, it is not a general solution that makes Boost or part of Boost a Cat1 library.
Now an example showing some dependencies:
MyCustomersApplication
|---- Boost 1.33
|---- MyLib
|---- Boost 1.32
If MyLib is Cat1 or Cat2 the chances are good that we may be ok. Most likely MyLib will soon adopt the migration to new Boost releases by the actions of the maintainer or by patches from end users. If however MyLib is Cat 3 or 4 we have a problem. This problem is real even if the use of Boost in MyLib is strictly internal. In my view this is an especially bad problem. Even if I do not intend to use Boost in any way in the API of MyLib I still have this issue of incompatibility.
Not if you maintain symbol isolation. Of course, that may not be possible for you.
Ok, back to the example. If MyLib have no intention of using Boost in the API, is there a way?
There are several ways to maintain symbol isolation. On Windows, it's fairly easy: simply use a shared library and don't dlexport anything involving boost. On *nix, recent GCCs can give you control over symbol visibility, so you can play the same kind of game. Otherwise -- and this is inconvenient at best -- you can do it by using dlopen to load a shared library containing the boost symbols, and speak to that library through one or more known entry points. That approach can be followed on Windows as well (using LoadLibrary) if symmetry with *nix turns out to be more appropriate than using the platform facilities.
Side Note:
I am aware of symbol encapsulation abilities in windows DLLs and similar new features supported by GCC, however these are not generally supported on all platforms, so I have selected to ignore them. They may however be helpful to many.
Okay, then you're stuck with the "plug-in" approach using dlopen.
The general idea is that there are no references to symbols in the boost namespace between any MyLib libraries, nor in any header files in or referenced from MyLib API. How can this be achieved? I have concluded that accessing boost trough namespace wrapping header files are unpractical at best. This needs to be designed into all of boost for this to work, and I do not know exactly how. The main challenge is #include <boost/xxxx.hpp> directives which does not refer to the wrapping headers.
Other libraries that allow that sort of thing have all of their headers #include both a prefix AND a suffix header. It would be a big change.
So to wrap up:
I do not believe anything prevents me from going along this path if I decide to do so. I am however interested in the boost developers attitude toward this approach, and in particular if this is something you will encourage and/or support. I am willing to work on the tool support for this in boost or to maintain a modified bcp tool if that is more appropriate.
I have mixed feelings about this. One of the things some people complain about when they look at the boost code is that the many workarounds get in the way of understanding. In fact, maintainability was one of the issues you listed at the beginning. So if we put in facilities for namespace renaming, our code would get uglier and less transparent. Maybe that's not a big deal, as we've already stepped off the "workaround cliff," but it is a concern for me. -- Dave Abrahams Boost Consulting www.boost-consulting.com
On Saturday 24 September 2005 01:09, David Abrahams wrote:
Bjørn Roald <bjorn@4roald.org> writes:
I have been working for some time to get acceptance for using Boost libraries as elements in a higher level portable platform for library development. It has surprised me how hard it is to make compelling and attractive set of arguments for using Boost (or other libraries) without attempting to ignore some pretty serious concerns. These concerns are:
- commercial support for our use of Boost
I know of a certain company offering commercial support for boost :)
Yes, I know about Boost Consulting. I did not mean to insinuate that there are no comercial support for Boost, just that it for many managers is important to have this kind of arrangement as part of a risk midigation in relation to open source. I expected you to react on the above, but left it anyway for completness, it also highlights that what you are doing is important. Also in face of licency issues it seems like it is hard for many to trust that you really are allowed to do what you are allowed to do with Boost or similar licences, for those none-beleivers it becommes easier to accept it if they acctually buy a product from somebody. I have no first hand experience in dealing with Boost Consulting, nether do I know anybody that have evaluated the company and their services. So I am in no possition to have an opinion about the quallity of what you can offer. It is certainly a positive sign that the 3 of you are so active on the developers list :-)
- quality assurance of Boost itself
Specifically?
Again, and as I mentioned to John, there are allways room for improvement, but I do think the boost developers do a lot. I bring this up because it is a genuine concern whenever you deal with sub-contractors or whatever, and it also apply to use of open source. Boost and or Boost Consulting could possibly make a more prominent reference to QA efforts and procedures on their web sites. At least have a bullet for it an tie together the procedures you have with peer reviews, testing, regression testing, bug tracking, change management, public developer mailing list. Then you can call this web page and its guidelines your Quallity Assuruance Plan.
- stability of Boost API
Yeah, there are no rules about it. However, we generally avoid breaking source compatibility.
If the library is good enough, this is a risk many will choose to live with. But it would be nice if library maintainers made ther _current_ view of how stable the API was public. Maybe on a scale used for all Boost libraries.
- maintainability of Boost source code
I actually find that most of the time it's pretty easy to dig into a Boost library and fix things,
good
although I seldom have to because the authors tend to be responsive to bug reports.
I have noticed that
However, there are no uniform standards for certain things that make maintenance easier for some people, like code formatting. But you probably had something else in mind?
Well I am pragmatic, we (programmers) will never agree on that stuff anyway ;-)
- deployability of our library into environments using different versions of Boost or libraries used in boost
Hmm, that's a tough one, especially if your library traffics in boost types in its public API, or if you use compilers that don't support symbol visibility control.
yes
- configuration management issues in general for us and our customers
Such as?
mostly a concern if we have to maintain other brances of boost than the library maintainers on boost.org.
- platform support in Boost ( or if the platform support Boost if you like ;-)
Sorry; what do you mean?
Just trying to be funny, sorry... the pun was that if boost is standard conformant, then the platforms are to blame. However, there are often some truth in what I try to joke about.
All of these concerns can be declared acceptable, given that you are willing to accept trade offs and risk. The issue I try to raze is if it is possible to reduce the magnitude of these risks and trade offs.
Good question.
One of the key issues is that you have to decide on some strategy of how to maintain your boost version. The most obvious strategy may be to stay with the latest Boost release at any time, this will however add risk as you have to adopt new changes as they are introduced into Boost and not as it fits your own release plans. The obvious alternative is to stay with a Boost release longer, possibly until you are forced to switch. This strategy will quickly force you into maintaining your own branch as Boost developers can not be expected to support old releases.
A company offering commercial Boost support can support old releases by integrating fixes into older release branches. Whether or not those branches can ever become "official Boost releases" is not clear, but even if they don't, at least you can share a common set of fixes with everyone else who is depending on that old release.
agree
Between these there are many shades of gray which may fit more or less well depending on your project and your customers. If your customers themselves use boost you are likely to meet a added configuration management, testing and release related cost and risk.
As library developers have a very different type of customer relationship than what is the case for end application developers there are a number of concerns that make adaptation of Boost, or any other 3rd party library for that sake, more problematic for library developers. The main issue is to what degree a library depends on other libraries and hence introduces indirect or direct dependencies to its users. These dependencies tend to be troublesome for the library developer to support, as well as they may be troublesome for users to accept. In fact users may easily be in situations where such dependencies are completely unacceptable. To help to understand the issue I like to define categories of libraries which affect the degree they are affected by these concerns.
Cat1. open source, source distributions, allowing encapsulation
Cat2. open source, source distributions in general
Cat3. open source, binary distributions
Cat4. closed source
Cat1 and Cat2 are build-able source code distributions.
I believe there are closed source libraries that are also delivered int the form of buildable source... but that's probably beside the point.
Depend of the definition of closed source, I consider it a Cat3 if it technically is buildable, but you have no licence to modify and then build. Often source code is open for debugging purposes only.
This give the user the ability to build the library and ensure that it is built correctly with other library it depends on, on the target system. Cat1 add to this the ability to hide the use of it inside other libraries, i.e. it can live concurrently in an end system with other versions of the same Cat1 library.
E.g. via symbol isolation or namespace renaming?
That is what we have in the toolbox, at least as far as I am aware of, yes.
Boost is a Cat2 library. Even if the extensive use of headers only libraries help some of the issues addressed in this post, it is not a general solution that makes Boost or part of Boost a Cat1 library.
Now an example showing some dependencies:
MyCustomersApplication
|---- Boost 1.33 | |---- MyLib | |---- Boost 1.32
If MyLib is Cat1 or Cat2 the chances are good that we may be ok. Most likely MyLib will soon adopt the migration to new Boost releases by the actions of the maintainer or by patches from end users. If however MyLib is Cat 3 or 4 we have a problem. This problem is real even if the use of Boost in MyLib is strictly internal. In my view this is an especially bad problem. Even if I do not intend to use Boost in any way in the API of MyLib I still have this issue of incompatibility.
Not if you maintain symbol isolation. Of course, that may not be possible for you.
I do not know how on all platforms I have to deal with.
Ok, back to the example. If MyLib have no intention of using Boost in the API, is there a way?
There are several ways to maintain symbol isolation. On Windows, it's fairly easy: simply use a shared library and don't dlexport anything involving boost. On *nix, recent GCCs can give you control over symbol visibility, so you can play the same kind of game. Otherwise -- and this is inconvenient at best -- you can do it by using dlopen to load a shared library containing the boost symbols, and speak to that library through one or more known entry points. That approach can be followed on Windows as well (using LoadLibrary) if symmetry with *nix turns out to be more appropriate than using the platform facilities.
Use of dlopen/dlsym and the win32 equallents is sertainly possible. Good point -- I will think more about that one. This could be appropriate if internal use of boost can be statically linked into a loadable library, and that module's services can be accessed through simple methods, alternatively through some framework - component hook-up. Interfaces to the module have to be Boost less.
Side Note:
I am aware of symbol encapsulation abilities in windows DLLs and similar new features supported by GCC, however these are not generally supported on all platforms, so I have selected to ignore them. They may however be helpful to many.
Okay, then you're stuck with the "plug-in" approach using dlopen.
yes, and renaming namespaces..
The general idea is that there are no references to symbols in the boost namespace between any MyLib libraries, nor in any header files in or referenced from MyLib API. How can this be achieved? I have concluded that accessing boost trough namespace wrapping header files are unpractical at best. This needs to be designed into all of boost for this to work, and I do not know exactly how. The main challenge is #include <boost/xxxx.hpp> directives which does not refer to the wrapping headers.
Other libraries that allow that sort of thing have all of their headers #include both a prefix AND a suffix header. It would be a big change.
Yes, and I do not like the prospect of that. David, you left out my text about use of bcp to rename the boost namespace here, that is what I wrap up.
So to wrap up:
I do not believe anything prevents me from going along this path if I decide to do so. I am however interested in the boost developers attitude toward this approach, and in particular if this is something you will encourage and/or support. I am willing to work on the tool support for this in boost or to maintain a modified bcp tool if that is more appropriate.
I have mixed feelings about this. One of the things some people complain about when they look at the boost code is that the many workarounds get in the way of understanding. In fact, maintainability was one of the issues you listed at the beginning. So if we put in facilities for namespace renaming, our code would get uglier and less transparent. Maybe that's not a big deal, as we've already stepped off the "workaround cliff," but it is a concern for me.
Ok, I agree with everything you write above, except the part about somebody falling off a cliff. But I feel like you missed one important point I tried to make. What I try to propose is something requiering no change in the way boost source code is written or maintained. I propose tool support for a _production_step_ ensuring namespace renaming in a subset of boost selected with the bcp tool. I would strongly advice all users of this tool support _not_ to maintain boost sources in the form where namespaces are renamed. I think of the original boost code as a model for generation of buildable source code. Kind of like CORBA IDL for idl2cpp compilers. Nowbody with their sences intact would modify the generated C++ binding code, at least not manually. -- Bjørn
bjorn@4roald.org writes:
On Saturday 24 September 2005 01:09, David Abrahams wrote:
Bjørn Roald <bjorn@4roald.org> writes:
- quality assurance of Boost itself
Specifically?
Again, and as I mentioned to John, there are allways room for improvement, but I do think the boost developers do a lot. I bring this up because it is a genuine concern whenever you deal with sub-contractors or whatever, and it also apply to use of open source.
I understand the notion of QA in general, but I want to know specifically what you think QA should look like, or look more like, for Boost.
Boost and or Boost Consulting could possibly make a more prominent reference to QA efforts and procedures on their web sites. At least have a bullet for it an tie together the procedures you have with peer reviews, testing, regression testing, bug tracking, change management, public developer mailing list. Then you can call this web page and its guidelines your Quallity Assuruance Plan.
Okay, that's a great idea. Boost Consulting's website desperately needs an overhaul anyway.
- stability of Boost API
Yeah, there are no rules about it. However, we generally avoid breaking source compatibility.
If the library is good enough, this is a risk many will choose to live with. But it would be nice if library maintainers made ther _current_ view of how stable the API was public. Maybe on a scale used for all Boost libraries.
Usually it's possible to announce imminent destabilization well in advance, so it would certainly be possible to have a status page that described those things.
- configuration management issues in general for us and our customers
Such as?
mostly a concern if we have to maintain other brances of boost than the library maintainers on boost.org.
Ah.
- platform support in Boost ( or if the platform support Boost if you like ;-)
Sorry; what do you mean?
Just trying to be funny, sorry... the pun was that if boost is standard conformant, then the platforms are to blame. However, there are often some truth in what I try to joke about.
I understand, but I don't even understand the first part. In what way is "platform support in Boost" a concern?
Ok, back to the example. If MyLib have no intention of using Boost in the API, is there a way?
There are several ways to maintain symbol isolation. On Windows, it's fairly easy: simply use a shared library and don't dlexport anything involving boost. On *nix, recent GCCs can give you control over symbol visibility, so you can play the same kind of game. Otherwise -- and this is inconvenient at best -- you can do it by using dlopen to load a shared library containing the boost symbols, and speak to that library through one or more known entry points. That approach can be followed on Windows as well (using LoadLibrary) if symmetry with *nix turns out to be more appropriate than using the platform facilities.
Use of dlopen/dlsym and the win32 equallents is sertainly possible. Good point -- I will think more about that one. This could be appropriate if internal use of boost can be statically linked into a loadable library, and that module's services can be accessed through simple methods, alternatively through some framework - component hook-up. Interfaces to the module have to be Boost less.
Yes. I can even imagine a nice tool or library for wrapping these interfaces up into isolated components so you and your clients don't have to deal with the entry point/type erasure nitty-gritty. That might be an excellent candidate for a Boost library that supports namespace renaming.
David, you left out my text about use of bcp to rename the boost namespace here, that is what I wrap up.
I didn't have much to say about it.
I do not believe anything prevents me from going along this path if I decide to do so. I am however interested in the boost developers attitude toward this approach, and in particular if this is something you will encourage and/or support. I am willing to work on the tool support for this in boost or to maintain a modified bcp tool if that is more appropriate.
I have mixed feelings about this. One of the things some people complain about when they look at the boost code is that the many workarounds get in the way of understanding. In fact, maintainability was one of the issues you listed at the beginning. So if we put in facilities for namespace renaming, our code would get uglier and less transparent. Maybe that's not a big deal, as we've already stepped off the "workaround cliff," but it is a concern for me.
Ok, I agree with everything you write above, except the part about somebody falling off a cliff.
:) I just meant that we're probably fully committed to having ugly workarounds, at least in some parts of Boost, so what does one more hurt?
But I feel like you missed one important point I tried to make. What I try to propose is something requiering no change in the way boost source code is written or maintained. I propose tool support for a _production_step_ ensuring namespace renaming in a subset of boost selected with the bcp tool.
Yes, I noticed that. It's a really neat idea, but I have no clue whether it's practical. Actually I do at least have a _clue_. Depending on the level of PP usage in a Boost header, it could be very difficult to identify all the places that symbols need to be changed. And even then a naive "replace the boost identifier" strategem is not guaranteed to work unless we all maintain an agreement that "boost" is never to be used as an identifier other than for the outer boost namespace. I'm not sure what other issues are lurking in there; these are just the two that popped into my head immediately. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Yes, I noticed that. It's a really neat idea, but I have no clue whether it's practical. Actually I do at least have a _clue_. Depending on the level of PP usage in a Boost header, it could be very difficult to identify all the places that symbols need to be changed. And even then a naive "replace the boost identifier" strategem is not guaranteed to work unless we all maintain an agreement that "boost" is never to be used as an identifier other than for the outer boost namespace. I'm not sure what other issues are lurking in there; these are just the two that popped into my head immediately.
I think what Bjorn's looking for may be possible: there are only a limited number of contexts in which a namespace name can occur, and I think we can locate and patch all of them with regexes, ignoring variations in whitespace they would be: namespace boost{ using namespace boost; boost:: I don't think there are any others? It's certainly true that this mechanism could be messed up by macro usage if you try hard enough, but I don't think we have any code that does that (I'm assuming that occurrence of these patterns *within* macro definitions would still get replaced). I guess the we'd just have to try it and see if any current libraries break this idiom. John.
John Maddock said:
I guess the we'd just have to try it and see if any current libraries break this idiom.
Should I post my code for bcp in the Vault or somewhere else? It includes both two new files and some changes to old ones. I will try some more my selves, will try to run test code as well. I know regexp but are no expert, so if I hit problems there I just post questions. I leave the ambition of support for nested namespaces until we know more. -- Bjørn
On Saturday 24 September 2005 21:38, Bjørn Roald wrote:
John Maddock said:
I guess the we'd just have to try it and see if any current libraries break this idiom.
I will try some more ...
I did some testing with my modified bcp tool, First I called bcp to copy some modules to /tmp/bcp and replace namespace boost with spoost "spooked boost" :) [bjorn@frodo boost]$ tools/bcp/run/bcp --cvs --namespace=spoost filesystem program_options boost-build.jam build Jamrules project-root.jam /tmp/bcp/ [bjorn@frodo debug]$ cd /tmp/bcp/ [bjorn@frodo bcp]$ ls -l total 44 drwxrwxr-x 3 bjorn bjorn 4096 Sep 25 23:47 bin -r--r--r-- 1 bjorn bjorn 482 Sep 25 23:44 boost-build.jam -r--r--r-- 1 bjorn bjorn 616 Sep 25 23:44 boost.css -r--r--r-- 1 bjorn bjorn 6308 Sep 25 23:44 boost.png -r--r--r-- 1 bjorn bjorn 435 Sep 25 23:44 Jamrules drwxrwxr-x 9 bjorn bjorn 4096 Sep 25 23:40 libs -r--r--r-- 1 bjorn bjorn 75 Sep 25 23:44 project-root.jam drwxrwxr-x 18 bjorn bjorn 4096 Sep 25 23:44 spoost drwxrwxr-x 3 bjorn bjorn 4096 Sep 25 23:45 stage drwxrwxr-x 3 bjorn bjorn 4096 Sep 25 23:44 tools I then compiled ./lib/program_options/build ./lib/program_options/test with plain "bjam" command in those directories no errors during build and test run as far as I saw. here are some checks I did on one of the test programs afterwords [bjorn@frodo bcp]$ cd /tmp/bcp/bin/boost/libs/program_options/test/parsers_test.test/gcc/debug [bjorn@frodo debug]$ ldd ./parsers_test libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00101000) libm.so.6 => /lib/tls/libm.so.6 (0x009b6000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0098e000) libc.so.6 => /lib/tls/libc.so.6 (0x00a69000) /lib/ld-linux.so.2 (0x0099c000) [bjorn@frodo debug]$ ./parsers_test Running 1 test case... *** No errors detected [bjorn@frodo debug]$ nm -C ./parsers_test | grep boost | wc 0 0 0 [bjorn@frodo debug]$ nm -C ./parsers_test | grep spoost | wc 2491 35631 785706 Note that there are no "boost" string in any binary symbols, while all 2491 boost symbols references are renamed "spoost" in binary. I also attached two source files so you can see the effect on the source. As far as I can see, I have successfully demonstrated feasability of an automated production line of boost library with modification to its namespace. A few questions before I go on to the next step: Is there a simple way of getting bcp to copy all of boost, i.e. without having to list all modules? And then; how do I run test on all of "spoost" and generate test reports that are suitable for evaluation? I guess there is a target in boost.build of some sort I can try. ----- Bjørn
As far as I can see, I have successfully demonstrated feasability of an automated production line of boost library with modification to its namespace.
Cool.
A few questions before I go on to the next step:
Is there a simple way of getting bcp to copy all of boost, i.e. without having to list all modules?
Yep, use "." as the module name.
And then; how do I run test on all of "spoost" and generate test reports that are suitable for evaluation? I guess there is a target in boost.build of some sort I can try.
Yep, use the Jamfile in /status. Suggest you run everything on the unmodified Boost, and then diff it with the result of the run on the modified version, or second thoughts just check that the number of failures (if any) is the same. I'm a bit caught up with the tr1 review at present, so if I become a little unresponce this week you know why :-/ John.
On Monday 26 September 2005 12:14, John Maddock wrote:
And then; how do I run test on all of "spoost" and generate test reports that are suitable for evaluation? I guess there is a target in boost.build of some sort I can try.
Yep, use the Jamfile in /status.
Suggest you run everything on the unmodified Boost, and then diff it with the result of the run on the modified version,
Did this, get 705 diffs, most comes as file paths and line number differences when samme warnings or errors are reported in both builds... :-) Thinking about trimming this before I do diff, and possibly running bcp without adding headers to files to get rid of line number diffs. But I have not tried that yet. There are stuff in the diff that looks like real problems, but I have not dug in yet.
or second thoughts just check that the number of failures (if any) is the same.
************ build log for unmodified boost code *************************** cd /dsk1/src/boost_1_33_0/status/ bjam --------------------------------------------------------------------- *** If you don't need Boost.Python, you can ignore this section *** *** pass --without-python to suppress this message in the future *** skipping Boost.Python library build due to missing or incorrect configuration couldn't find Python.h in "/usr/local/include/python2.2" You can configure the location of your python installation by setting: PYTHON_ROOT - currently "/usr/local" PYTHON_VERSION - The 2-part python Major.Minor version number (e.g. "2.2", NOT "2.2.1") - currently "2.2" The following are automatically configured from PYTHON_ROOT if not otherwise set: PYTHON_LIB_PATH - path to Python library object; currently "/usr/local/lib/python2.2/config" PYTHON_INCLUDES - path to Python #include directories; currently "/usr/local/include/python2.2" --------------------------------------------------------------------- Building Boost.Regex with the optional Unicode/ICU support disabled. Please refer to the Boost.Regex documentation for more information (and if you don't know what ICU is then you probably don't need it). ...patience... ...patience... ...patience... ...patience... ...patience... ...found 37791 targets... ...updating 9832 targets... MkDir1 ../bin MkDir1 ../bin/boost MkDir1 ../bin/boost/libs MkDir1 ../bin/boost/libs/test MkDir1 ../bin/boost/libs/test/build MkDir1 ../bin/boost/libs/test/build/libboost_test_exec_monitor.a MkDir1 ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc MkDir1 ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/compiler_log_formatter.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/execution_monitor.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/framework.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/plain_report_formatter.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/progress_monitor.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/results_collector.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/results_reporter.o ************* a lot of stuff cut out **************************** ...failed RmTemps ../bin/boost/libs/test/test/minimal_test.test/gcc/debug/minimal_test.run... execute-test ../bin/boost/libs/test/test/online_test.test/gcc/debug/online_test.run **passed** ../bin/boost/libs/test/test/online_test.test/gcc/debug/online_test.test execute-test ../bin/boost/libs/test/test/errors_handling_test.test/gcc/debug/errors_handling_test.run **passed** ../bin/boost/libs/test/test/errors_handling_test.test/gcc/debug/errors_handling_test.test **passed** ../bin/boost/status/timer_test.test/gcc/debug/timer_test.test execute-test ../bin/boost/status/simple_example_5.test/gcc/debug/simple_example_5.run **passed** ../bin/boost/status/simple_example_5.test/gcc/debug/simple_example_5.test execute-test ../bin/boost/status/simple_example_4.test/gcc/debug/simple_example_4.run **passed** ../bin/boost/status/simple_example_4.test/gcc/debug/simple_example_4.test execute-test ../bin/boost/status/simple_example_3.test/gcc/debug/simple_example_3.run **passed** ../bin/boost/status/simple_example_3.test/gcc/debug/simple_example_3.test execute-test ../bin/boost/status/simple_example_2.test/gcc/debug/simple_example_2.run **passed** ../bin/boost/status/simple_example_2.test/gcc/debug/simple_example_2.test execute-test ../bin/boost/status/simple_example_1.test/gcc/debug/simple_example_1.run **passed** ../bin/boost/status/simple_example_1.test/gcc/debug/simple_example_1.test execute-test ../bin/boost/status/examples.test/gcc/debug/examples.run ...on 9800th target... **passed** ../bin/boost/status/examples.test/gcc/debug/examples.test execute-test ../bin/boost/status/io_test.test/gcc/debug/io_test.run **passed** ../bin/boost/status/io_test.test/gcc/debug/io_test.test execute-test ../bin/boost/status/tuple_test_bench.test/gcc/debug/tuple_test_bench.run **passed** ../bin/boost/status/tuple_test_bench.test/gcc/debug/tuple_test_bench.test ...failed updating 19 targets... ...skipped 28 targets... ...updated 9785 targets... Compilation exited abnormally with code 1 at Wed Sep 28 02:29:57 ******************* end build log *************************************' ************ Build log for modified spoost code ********************* cd /tmp/bcp/spoost/status/ bjam --------------------------------------------------------------------- *** If you don't need Boost.Python, you can ignore this section *** *** pass --without-python to suppress this message in the future *** skipping Boost.Python library build due to missing or incorrect configuration couldn't find Python.h in "/usr/local/include/python2.2" You can configure the location of your python installation by setting: PYTHON_ROOT - currently "/usr/local" PYTHON_VERSION - The 2-part python Major.Minor version number (e.g. "2.2", NOT "2.2.1") - currently "2.2" The following are automatically configured from PYTHON_ROOT if not otherwise set: PYTHON_LIB_PATH - path to Python library object; currently "/usr/local/lib/python2.2/config" PYTHON_INCLUDES - path to Python #include directories; currently "/usr/local/include/python2.2" --------------------------------------------------------------------- Building Boost.Regex with the optional Unicode/ICU support disabled. Please refer to the Boost.Regex documentation for more information (and if you don't know what ICU is then you probably don't need it). ...patience... ...patience... ...patience... ...patience... ...patience... ...found 37780 targets... ...updating 9832 targets... MkDir1 ../bin MkDir1 ../bin/boost MkDir1 ../bin/boost/libs MkDir1 ../bin/boost/libs/test MkDir1 ../bin/boost/libs/test/build MkDir1 ../bin/boost/libs/test/build/libboost_test_exec_monitor.a MkDir1 ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc MkDir1 ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/compiler_log_formatter.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/execution_monitor.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/framework.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/plain_report_formatter.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/progress_monitor.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/results_collector.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/results_reporter.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/test_main.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/test_tools.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/unit_test_log.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/unit_test_main.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/unit_test_monitor.o gcc-C++-action ../bin/boost/libs/test/build/libboost_test_exec_monitor.a/gcc/debug/unit_test_parameters.o ************* a lot of stuff cut out **************************** if test $verbose -eq 0 ; then echo ====== BEGIN OUTPUT ====== cat ../bin/boost/libs/test/test/errors_handling_test.test/gcc/debug/errors_handling_test.output echo ====== END OUTPUT ====== fi exit $status ...failed execute-test ../bin/boost/libs/test/test/errors_handling_test.test/gcc/debug/errors_handling_test.run... **passed** ../bin/boost/status/timer_test.test/gcc/debug/timer_test.test execute-test ../bin/boost/status/simple_example_5.test/gcc/debug/simple_example_5.run **passed** ../bin/boost/status/simple_example_5.test/gcc/debug/simple_example_5.test execute-test ../bin/boost/status/simple_example_4.test/gcc/debug/simple_example_4.run **passed** ../bin/boost/status/simple_example_4.test/gcc/debug/simple_example_4.test execute-test ../bin/boost/status/simple_example_3.test/gcc/debug/simple_example_3.run **passed** ../bin/boost/status/simple_example_3.test/gcc/debug/simple_example_3.test execute-test ../bin/boost/status/simple_example_2.test/gcc/debug/simple_example_2.run **passed** ../bin/boost/status/simple_example_2.test/gcc/debug/simple_example_2.test execute-test ../bin/boost/status/simple_example_1.test/gcc/debug/simple_example_1.run **passed** ../bin/boost/status/simple_example_1.test/gcc/debug/simple_example_1.test execute-test ../bin/boost/status/examples.test/gcc/debug/examples.run **passed** ../bin/boost/status/examples.test/gcc/debug/examples.test execute-test ../bin/boost/status/io_test.test/gcc/debug/io_test.run **passed** ../bin/boost/status/io_test.test/gcc/debug/io_test.test execute-test ../bin/boost/status/tuple_test_bench.test/gcc/debug/tuple_test_bench.run **passed** ../bin/boost/status/tuple_test_bench.test/gcc/debug/tuple_test_bench.test ...failed updating 33 targets... ...skipped 49 targets... ...updated 9750 targets... Compilation exited abnormally with code 1 at Wed Sep 28 11:50:50 ...failed updating 19 targets... ...skipped 28 targets... ...updated 9785 targets... ******************* end build log *************************************' note that the boost build had 37791 targets while spoost had 37780 37791-37780=11 Any idea why? Be aware that I used bcp without the --cvs switch since I downloaded a clean 1.33.0 tar.gz package for the test - no CVS info in that download. RESULT SUMMARY: targets failed skipped updated boost 37791 19 28 9785 spoost 37780 33 49 9750 diff 11 14 21 35 I have the log files if you are interested in details, but they are large so I don't want to post them in the list. -rw-rw-r-- 1 bjorn bjorn 3513039 Sep 28 08:53 bcp.log -rw-rw-r-- 1 bjorn bjorn 1375140 Sep 28 08:27 boost_build.log -rw-rw-r-- 1 bjorn bjorn 2113028 Sep 28 20:57 spoost_build.log Note how much bigger the spoost_build.log is. There are definently some long error listings in there ;-\ Maybe I should compile a list of the targets that failed, boost and spoost side-by-side. If I post that on the list it may be that somebody see a pattern that may be helpful. Next step anyways is for me to dig in and find the problems. One thought, would it be an idea if bcp was modified so it only replaces files if it actually detect a difference from the file which is going to be replaced on the disk? I have done that with some other code generators, and it is helpfull in combination with incremental build capabilities to avoid rebuilding too much too often. It could be turned on by a command line switch. This may be important if it is not going to be tempting to start modifying the generated source, to save build time.
I'm a bit caught up with the tr1 review at present, so if I become a little unresponce this week you know why :-/
No problem, this is very much a spare time activity for me. No real rush :-) I really appreciate your help John, thanks. Bjørn
On Saturday 24 September 2005 12:51, David Abrahams wrote:
bjorn@4roald.org writes:
On Saturday 24 September 2005 01:09, David Abrahams wrote:
Bjørn Roald <bjorn@4roald.org> writes:
- configuration management issues in general for us and our customers
Such as?
mostly a concern if we have to maintain other brances of boost than the library maintainers on boost.org.
Ah.
I may have been a little quick here. It really concerns me that I sell a library were I have to send with it a description with a long list of libraries-versions my customers configuration management will have to deal with. One of the items on the long list is me telling the customers configuration manager to ensure that if they use boost directly or indirectly through other libraries, they better use 1.32.0 compiled with the following compiler std lib and options.... Configuration Nightmare -- and it is assured to come back to me and haunt me. My library may have to support many versions of boost, with potential API changes. Ideally the list should be empty, at least as short as possible.
- platform support in Boost ( or if the platform support Boost if you like ;-)
Sorry; what do you mean?
Just trying to be funny, sorry... the pun was that if boost is standard conformant, then the platforms are to blame. However, there are often some truth in what I try to joke about.
I understand, but I don't even understand the first part. In what way is "platform support in Boost" a concern?
Sun compilers come to mind. I have had to massage boost code every time I used it with sun compilers. And I have to admit I usually avoid trying :-( I believe Steve Clamage of Sun recently announced that the Sun CC would be boost compatible soon. Anybody have an update on that? I have not tried with any of the newest versions of SunStudio and I don't believe they are in the regression tests either. Also, I believe ACE have a much longer bragging list when it comes to platforms supported. I am not saying the portability is bad, just that it still may be a concern for some.
Use of dlopen/dlsym and the win32 equallents is sertainly possible. Good point -- I will think more about that one. This could be appropriate if internal use of boost can be statically linked into a loadable library, and that module's services can be accessed through simple methods, alternatively through some framework - component hook-up. Interfaces to the module have to be Boost less.
Yes. I can even imagine a nice tool or library for wrapping these interfaces up into isolated components so you and your clients don't have to deal with the entry point/type erasure nitty-gritty. That might be an excellent candidate for a Boost library that supports namespace renaming.
Funny, I have a colleague battering me about the same general idea in a completely different context. This is part of some work we do on a component framework. His approach was using some pretty advanced macro stuff, and I am always skeptical to that, probably to skeptical some times. He is a Java Guru really, and relatively new to C++. He thought WOW preprocessor -- powerful. I am trying to moderate, and point at other features in the language. But there may be more to this, I need to pay more attention :) Are you thinking about connecting this to some sort of new library abstraction feature in form of a library? And possibly combined with something generating code based on macros, templates or just some other tool/model/pattern? How does this tie to work on the Boost Interface Library proposal?
I just meant that we're probably fully committed to having ugly workarounds, at least in some parts of Boost, so what does one more hurt?
While cross country skiing in the Norwegian mountains we have a saying: "It is never too late to turn around". It refers to the prospect of bad weather ahead.
But I feel like you missed one important point I tried to make. What I try to propose is something requiering no change in the way boost source code is written or maintained. I propose tool support for a _production_step_ ensuring namespace renaming in a subset of boost selected with the bcp tool.
Yes, I noticed that. It's a really neat idea, but I have no clue whether it's practical. Actually I do at least have a _clue_. Depending on the level of PP usage in a Boost header, it could be very difficult to identify all the places that symbols need to be changed. And even then a naive "replace the boost identifier" strategem is not guaranteed to work unless we all maintain an agreement that "boost" is never to be used as an identifier other than for the outer boost namespace. I'm not sure what other issues are lurking in there; these are just the two that popped into my head immediately.
Well you are right, in the tests I have done I quickly I had to add special treatment for "boost.org" in the license. There will be more. ---- Bjørn
Bjørn Roald <bjorn@4roald.org> writes:
On Saturday 24 September 2005 12:51, David Abrahams wrote:
I can even imagine a nice tool or library for wrapping these interfaces up into isolated components so you and your clients don't have to deal with the entry point/type erasure nitty-gritty. That might be an excellent candidate for a Boost library that supports namespace renaming.
Are you thinking about connecting this to some sort of new library abstraction feature in form of a library? And possibly combined with something generating code based on macros, templates or just some other tool/model/pattern?
Well, it's just a vague idea at this point, but if you're interested in developing it you can always contact me offline ;-)
How does this tie to work on the Boost Interface Library proposal?
I don't know; that certainly does seem like it might be useful in the endeavor I'm describing. -- Dave Abrahams Boost Consulting www.boost-consulting.com
On Sat, 24 Sep 2005 11:51:42 +0100, David Abrahams <dave@boost-consulting.com> wrote:
Again, and as I mentioned to John, there are allways room for improvement, but I do think the boost developers do a lot. I bring this up because it is a genuine concern whenever you deal with sub-contractors or whatever, and it also apply to use of open source.
I understand the notion of QA in general, but I want to know specifically what you think QA should look like, or look more like, for Boost.
Boost and or Boost Consulting could possibly make a more prominent reference to QA efforts and procedures on their web sites. At least have a bullet for it an tie together the procedures you have with peer reviews, testing, regression testing, bug tracking, change management, public developer mailing list. Then you can call this web page and its guidelines your Quallity Assuruance Plan.
Okay, that's a great idea. Boost Consulting's website desperately needs an overhaul anyway.
I'm going to need to do a formal assessment of the 'quality' of Boost libraries and so this caught my eye. I did some more reading on the Boost web site about the review procedures, code guidelines, etc., etc. It seems to me that Boost has its procedures in place and documented and what would help someone assessing Boost is a higher profile for them. So I'd support the idea of a page called something like Quality Assurance Plan. Perhaps a section for library development and one on Boost releases with links to the existing guidelines. One thing I don't see documented is how review managers are selected (beyond there being a queue which the Review Wizard manages). The peer review allows anyone to submit comments and it's up to the review manager and library author to consider them. How do we know that the review manager has suitable experience of Boost and C++ (or whatever) to carry out his/her role? Ultimately it seems that the quality of Boost rests on the experience of its library authors and reviewers, so how does an observer assess that? Richard
On Sep 26, 2005, at 8:48 AM, Richard Jennings wrote:
So I'd support the idea of a page called something like Quality Assurance Plan. Perhaps a section for library development and one on Boost releases with links to the existing guidelines.
This would be interesting.
One thing I don't see documented is how review managers are selected (beyond there being a queue which the Review Wizard manages). The peer review allows anyone to submit comments and it's up to the review manager and library author to consider them. How do we know that the review manager has suitable experience of Boost and C++ (or whatever) to carry out his/her role?
Review manages are long-time, established members of the Boost community and are almost exclusively authors of existing Boost libraries. They are implicitly trusted by the Boost community to make sound decisions.
Ultimately it seems that the quality of Boost rests on the experience of its library authors and reviewers, so how does an observer assess that?
Given that all library authors and reviewers are volunteers, I'm not sure how we can assess them as a group. Doug
On Mon, 26 Sep 2005 17:58:51 +0100, Douglas Gregor <doug.gregor@gmail.com> wrote:
Review manages are long-time, established members of the Boost community and are almost exclusively authors of existing Boost libraries. They are implicitly trusted by the Boost community to make sound decisions.
Ultimately it seems that the quality of Boost rests on the experience of its library authors and reviewers, so how does an observer assess that?
Given that all library authors and reviewers are volunteers, I'm not sure how we can assess them as a group.
Thanks for the clarification on Review Managers. Where I am coming from is that if you are following an ISO9001-style quality system then you (I) need to assess off the shelf (OTS) software before using it. This includes commercial, shrink-wrapped and open source software. To do this you would assess the company/organisation producing the software as well as the softare. In a commercial company you might expect there to be training records that show that a person is suitably trained and experienced to do their job (you can debate whether training records actually show this or not). I guess the ideal would be if the biographies of Boost People included more about technical experience and contribution to Boost (given that this is one of your criteria for being a Review Manager). I accept that people may not want to provide this information. How about a citations page that lists glowing references to Boost? The Who's Using Boost page also backs up the wide use and acceptance of Boost. I hope it's clear that I'm _not_ questioning the quality of Boost. I am just musing how I would prove to a Food and Drug Administration Inspector that Boost is indeed "...one of the most highly regarded and expertly designed C++ library projects in the world". Richard
As someone who has suffered from ISO 900x and the (threat of) FDA inspections, I think these are a very useful, and cheap, suggestions. I think it will make is MUCH easier for regulated organisations to use Boost, especially as Boost exists only in cyberspace and is thus viewed with DEEP suspicion. You might also add a bullet point for support and regular corrections and improvements, on which we have a FAR better record than most commercial organisations. You can't kick any butts or sue anyone, but you are actually more likely to get effective help. Paul Paul A Bristow Prizet Farmhouse, Kendal, Cumbria UK LA8 8AB +44 1539 561830 +44 7714 330204 mailto: pbristow@hetp.u-net.com www.hetp.u-net.com | -----Original Message----- | From: boost-bounces@lists.boost.org | [mailto:boost-bounces@lists.boost.org] On Behalf Of Richard Jennings | Sent: 28 September 2005 10:52 | To: boost@lists.boost.org | Subject: Re: [boost] Quality assurance (was Re: [tools][bcp] | using boostinternally in other libraries) | | On Mon, 26 Sep 2005 17:58:51 +0100, Douglas Gregor | <doug.gregor@gmail.com> | wrote: | | > Review manages are long-time, established members of the Boost | > community and are almost exclusively authors of existing Boost | > libraries. They are implicitly trusted by the Boost | community to make | > sound decisions. | > | >> Ultimately it seems that the quality of Boost rests on the | >> experience of its | >> library authors and reviewers, so how does an observer assess that? | > | > Given that all library authors and reviewers are volunteers, I'm not | > sure how we can assess them as a group. | > | | Thanks for the clarification on Review Managers. | | Where I am coming from is that if you are following an ISO9001-style | quality system then you (I) need to assess off the shelf | (OTS) software | before using it. This includes commercial, shrink-wrapped and open | source software. To do this you would assess the company/organisation | producing the software as well as the softare. In a | commercial company | you might expect there to be training records that show that a person | is suitably trained and experienced to do their job (you can debate | whether training records actually show this or not). | | I guess the ideal would be if the biographies of Boost People included | more about technical experience and contribution to Boost (given that | this is one of your criteria for being a Review Manager). I accept | that people may not want to provide this information. | | How about a citations page that lists glowing references to Boost? | The Who's Using Boost page also backs up the wide use and acceptance | of Boost. | | I hope it's clear that I'm _not_ questioning the quality of Boost. I | am just musing how I would prove to a Food and Drug Administration | Inspector that Boost is indeed "...one of the most highly regarded | and expertly designed C++ library projects in the world". | | Richard
As a taster to what I had in mind for the Qualilty Assurance Plan I have put up a couple of pages on the Boost Wiki with a rough outline: http://tinyurl.com/e4sxw. The one on Boost Product is more developed than the other. Before I go too much further with my suggestions I wanted to gauge interest. Just to reiterate that the aim is to "pull together" the existing policies and guidelines rather than make new policy. Richard
"Richard Jennings" <richard.jennings@teraview.com> writes:
As a taster to what I had in mind for the Qualilty Assurance Plan I have put up a couple of pages on the Boost Wiki with a rough outline: http://tinyurl.com/e4sxw. The one on Boost Product is more developed than the other.
Before I go too much further with my suggestions I wanted to gauge interest.
Just to reiterate that the aim is to "pull together" the existing policies and guidelines rather than make new policy.
Richard, you and Paul seem to be the experts in this area. I have no personal interest in these pages, but Boost in general has an interest in making sure our libraries are broadly usable. If you (and Paul, for example) want to drive an effort to put together the documentation needed, you'd have my support. -- Dave Abrahams Boost Consulting www.boost-consulting.com
On 10/03/2005 09:01 AM, Richard Jennings wrote:
As a taster to what I had in mind for the Qualilty Assurance Plan I have put up a couple of pages on the Boost Wiki with a rough outline: http://tinyurl.com/e4sxw. The one on Boost Product is more developed than the other. BoostLibraryQualityPlan has:
* User Requirements * Technical Requirements Is the first more or less a user's guide and the 2nd more or less a maintainer's guide (i.e. enables someone to understand how the library works and thereby easy modifications)?
... Trust and judgement. One useful input, for those that can judge it is, IMHO: http://www.boost.org/people/people.htm -- Alec Ross
[note to mods: apologies, please kill my earlier post on this. It was over aggressive in removal of context.]
Ultimately it seems that the quality of Boost rests on the experience of its library authors and reviewers, so how does an observer assess that?
Trust and judgement. One useful input, for those that can judge it is, IMHO: http://www.boost.org/people/people.htm -- Alec Ross
participants (9)
-
Alec Ross -
bjorn＠4roald.org -
Bjørn Roald
-
David Abrahams -
Douglas Gregor -
John Maddock -
Larry Evans -
Paul A Bristow -
Richard Jennings