[Booster] Or boost is useless for library developers
Hello, I want to bring up an important issue that boost developers are not aware of: anybody who develops a library and wants to keep any kind of backward compatibility can-not use Boost. Issue, let's assume I want to develop a small library that keeps backward comparability well. Let's assume I want to use `boost::shared_ptr` as one of the most important parts of boost in my library interface... Impossible. why? Even, for such a small part of boost with stable (even tr1) definitions it is impossible to promise that it would not be broken between releases. Why boost::details::sp_counted_base may be changed because in next version atomic counter was implemented using assembly rather then pthread's mutex... which affects the layout of boost::details::sp_counted_base and thus breaks compatibility. That is not a first time I bring up this issue. But at this point I started to bring a some kind of solution. I do need boost for my FOSS project CppCMS <http://cppcms.sf.net>. I was trying for a long period use renamed version of Boost (with a small tool I written similar to BCP's renamer) inside the library not showing the interface outside but finally I realized that I need boost in interface. I need ASIO, I need shared_ptr, I need regex, I need thread and much more... So I created a small library I called "booster" as part of CppCMS project that has a goal - provide backward compatible and stable boost like library that can be used by other libraries. It has following libraries: - smart_ptr: intrusive_ptr, shared_ptr, - function - locale (full copy of Boost.Locale I develop) - regex (pcre-wrapper, no replace functionality, no wide api) - system (meanwhile without error_condition) - thread (partial) - aio - my own ASIO like library. - ptime - Posix time (very different from boost one) For this purposes I did: - Adopted some of boost code to booster (shared_ptr/intrusive_ptr) and made sure that it would not break ABI with internal changes. - Wrapped pthreads with boost.thread like interface (in any case boost.thread is more like pthreads with C++ interface). So under windows it requires pthreads-win32 library. - I wrapped pcre library with boost.regex like interface. - I had written my own Boost.Asio like library (Booster.AIO) that has very limited use of templates. It uses epoll/devpoll/kqueue where available and poll or select on other platforms (no IOCP as it adds many limitations). - It uses CMake instead of Boost.Build. - It supports **only** modern compilers: gcc>=3.4, MSVC-2008 (2005 should work as well, but not tested), intel 11 compiler. - It is little bit less Windows friendly then Boost - no auto-link, no wide api, no IOCP support. Believe me it wasn't simple decision to start writing this library. But I just had no other choice. ------------------------------------------------ The source code can be found there (svn): https://cppcms.svn.sourceforge.net/svnroot/cppcms/framework/branches/refacto... - I do not encourage anybody use it over boost, but if you have similar problems with boost as I have, you are welcome to take a look on it, try it. - I do think that boost is great library. - I would really prefer to use boost if it was not breaking ABI every few month. - I would probably adopt more boost libraries into booster (where possible) ------------------------------------------------------------- I want boost community to think about it... Is it right that developers of many libraries can't use Boost. Think of GTKmm, think of Qt, think of kdelib - that no-one can use it just because they promise backward compatibility. Maybe there is something wrong with it? Artyom.
Hello,
Let's assume I want to use `boost::shared_ptr` as one of the most important parts of boost in my library interface...
Impossible. why?
Even, for such a small part of boost with stable (even tr1) definitions it is impossible to promise that it would not be broken between releases.
Why boost::details::sp_counted_base may be changed because in next version atomic counter was implemented using assembly rather then pthread's mutex... which affects the layout of boost::details::sp_counted_base and thus breaks compatibility.
This is solely about binary interface compatibility, right? I dare to say that this is a tradeoff that is accepted by adopting the template-based design. Boost calls itself a template library, after all. Inside a library or application, this is exactly what I ask for: C++ allows me to trade stable binary interfaces for more in-depth optimization possibilities, and I see boost as a valuable community effort which explores these possibilities. As soon as library boundaries are reached and one wants to care about binary compatibility, boost might just not be the right tool for the job. But why not using what boost has to offer inside the library, and approach the library ABI problem separately? What I find more problematic is the mix of header-only and compiled libraries boost has: If I would use a purely header-only template library for writing libraries and built the interfaces independently from the changing parts of the template library, I could decide for each library separately if I want to compile it against an upgraded template library or rather keep an old version. As soon as I do this with boost and start to use libraries with compiled parts, I have to keep them all in sync with the installed compiled libraries. But apart from this issue, I'd say boost is very good at what it does. And maybe it's even possible to approach the ABI problem in a dedicated library. For example a boost::abi which strips out all the juicy parts and is able to expose data in some kind of an abstract virtual method based interface. This comes with some overhead, but then again it's only at the library interface, while some libraries have this overhead in all possible places. Or maybe boost::corba... As far as I understand your effort, you are more or less taking boost and try to keep the binary layout stable. Personally, I'd fear to get stuck on less than optimal design decisions that the official boost can simply abandon because it is not bound to binary compatibility. So I feel more comfortable with tackling the interface problem separately.
- It uses CMake instead of Boost.Build.
Let me also do some advertising :) You might be interested in my recent effort to resurrect boost-cmake: [1]
Think of GTKmm, think of Qt, think of kdelib - that no-one can use it just because they promise backward compatibility.
Well, to achieve this I see for example Qt reinventing about every bit of STL. This is a way to ensure binary compatibility, but is it the right way to go? Best regards, Isidor [1] http://permalink.gmane.org/gmane.comp.lib.boost.cmake/833
This is solely about binary interface compatibility, right?
Yes.
I dare to say that this is a tradeoff that is accepted by adopting the template-based design. Boost calls itself a template library, after all.
Not exactly... Many if not most libraries consists of template part and non-template part.
<snip> As soon as library boundaries are reached and one wants to care about binary compatibility, boost might just not be the right tool for the job.
The problem that for C++ today is **THE** library. It gives C++ what JDK gives to Java, standard Python library gives to Python and so on. It is very good to have standard libraries but if your standard library breaks compatibility each few month it significantly limits it's target audience.
But why not using what boost has to offer inside the library, and approach the library ABI problem separately?
In CppCMS project I have two parts: 1. Public booster -- boost like libraries I need in the interface. 2. Private (cppcms_boost) -- renamed part of boost that is linked statically to my library and has it's own namespace so it would not collide with ordinary boost. So I would like to see something like boost::stable::* where interfaces keep binary backward compatibility.
What I find more problematic is the mix of header-only and compiled libraries <snip>
It is same problem. Let's see I have library foo compiled against header only part of boost-1.55 and it uses shared_ptr in it's interface. Now come used that uses foo with boost-1.56 with different binary interface (let's tell mutex replaced with atomic-ops) Oooops - your application is not thread safe. BIG Problem.
But apart from this issue, I'd say boost is very good at what it does. And maybe it's even possible to approach the ABI problem in a dedicated library. For example a boost::abi which strips out all the juicy parts
I think **anything** in tr1 can be binary stabilized and this is huge and most important part of Boost. Also many others can do the same. Of course I do not expect any library to be binary stable but I do expect to have some subset of boost libraries to be released separately as Boost.Stable
As far as I understand your effort, you are more or less taking boost and try to keep the binary layout stable. Personally, I'd fear to get stuck on less than optimal design decisions that the official boost can simply abandon because it is not bound to binary compatibility. So I feel more comfortable with tackling the interface problem separately.
Yes, it is problem of any library not connected to binary on programming interface. So you cleanup it between major releases once in several years.
- It uses CMake instead of Boost.Build.
Let me also do some advertising :)
You might be interested in my recent effort to resurrect boost-cmake: [1]
I would be glad if CMake had replaced boost-build...
Well, to achieve this I see for example Qt reinventing about every bit of STL. This is a way to ensure binary compatibility, but is it the right way to go?
Qt is only once example, And BTW qt works quite well with STL... But take a look on GTKmm? Bad library? And many others. The problem is that I am as modern library developer limited **not-to-use** boost **at-all** if I want keep ABI backward compatible. Artyom
Artyom wrote:
This is solely about binary interface compatibility, right ?
Yes.
Well, due to the inner shape of boost, such change are inevitable. What can be done is having such ABI-breaking change documented in a big, red, blinking part of the documentation. Did you repackaged libstd++5 when the 6 went out and broke some binary compatibility ? -- ___________________________________________ Joel Falcou - Assistant Professor PARALL Team - LRI - Universite Paris Sud XI Tel : (+33)1 69 15 66 35
Well, due to the inner shape of boost, such change are inevitable. What can be done is having such ABI-breaking change documented in a big, red, blinking part of the documentation.
Then all documentation would be red and blinking. Two reasons: - Class layout of many libraries may change with some small feature in configuration that may depend if you compile with gcc-4.1 or gcc-4.3 - Good example shared_ptr: - You may add debug define and class layout is changed - Sometimes it uses pthread sometimes atomic operations and it depends on so many things. And every 2nd boost upgrade added implementation for new platform. So **every library** that uses shared_ptr (like every 2nd boost library) would break ABI. The problem that Boost as whole never looks on ABI.
Did you repackaged libstd++5 when the 6 went out and broke some binary compatibility ?
6 Years ago? To be honest these days I don't remember if I had any project in C++ at all ;-) Artyom
On 5/15/2010 9:32 AM, Artyom wrote:
Did you repackaged libstd++5 when the 6 went out and broke some binary compatibility ?
6 Years ago? To be honest these days I don't remember if I had any project in C++ at all ;-)
A more apt question would be.. What did you do for MSVC CRT5=>CRT6=>CRT7=>CRT8. Those change almost every year. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org (msn) - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail
A more apt question would be.. What did you do for MSVC CRT5=>CRT6=>CRT7=>CRT8. Those change almost every year.
MSVC6 1998 MSVC7 2002 - 4 years MSVC8 2005 - 3 years MSVC9 2008 - 3 years MSVC10 2010 - 2 years Not so good but not so bad in comparison to GCC's 6 years or Boost's every 3 month! In any case for my specific project I would my users use Linux, FreeBSD, OpenSolaris :-) Artyom
adopting the template-based design. Boost calls itself a template library, after all.
Not exactly... Many if not most libraries consists of template part and non-template part.
Well, when I started using boost, I limited myself on the header-only ones so I could deploy the application code on systems without boost installed by simply bundling the headers, without having to bother with integrating the build system. There was still much useful functionality to get out of boost.
<snip> As soon as library boundaries are reached and one wants to care about binary compatibility, boost might just not be the right tool for the job.
The problem that for C++ today is **THE** library. It gives C++ what JDK gives to Java, standard Python library gives to Python and so on.
It is very good to have standard libraries but if your standard library breaks compatibility each few month it significantly limits it's target audience.
I'd say the audience which appreciates state-of-the-art C++ programming paradigms as found in boost _is_ significantly different from the audience more traditional programming paradigms have. And it seems like this audience actually sees the things that make boost unique as an advantage.
But why not using what boost has to offer inside the library, and approach the library ABI problem separately?
In CppCMS project I have two parts:
1. Public booster -- boost like libraries I need in the interface. 2. Private (cppcms_boost) -- renamed part of boost that is linked statically to my library and has it's own namespace so it would not collide with ordinary boost.
So I would like to see something like boost::stable::* where interfaces keep binary backward compatibility.
I hear you. I'm just saying that in the cases where I need a stable ABI, I rather go for interfacing glue than for a library fork.
What I find more problematic is the mix of header-only and compiled libraries <snip>
It is same problem. Let's see I have library foo compiled against header only part of boost-1.55 and it uses shared_ptr in it's interface.
Now come used that uses foo with boost-1.56 with different binary interface (let's tell mutex replaced with atomic-ops)
I was referring to the case where I keep the volatile boost code completely out of the interface. In this case I do not have the problem you are referring to, but I can get a problem with boost's shared libraries.
But apart from this issue, I'd say boost is very good at what it does. And maybe it's even possible to approach the ABI problem in a dedicated library. For example a boost::abi which strips out all the juicy parts
I think **anything** in tr1 can be binary stabilized and this is huge and most important part of Boost.
Sure, it can be done. But it would sacrifice the huge possibilities for future improvements which are possible by limiting to source compatibility.
- It uses CMake instead of Boost.Build.
Let me also do some advertising :)
You might be interested in my recent effort to resurrect boost-cmake: [1]
I would be glad if CMake had replaced boost-build...
I fully agree. But for now, I'd be even grateful if CMake found its way back into the official boost tree. And I believe this is more likely if the existing CMake-based boost codebase is up-to-date.
Well, to achieve this I see for example Qt reinventing about every bit of STL. This is a way to ensure binary compatibility, but is it the right way to go?
Qt is only once example, And BTW qt works quite well with STL...
It works with STL, but it still provides its own replacements for most of STL. This is why I immediately believe if they claim they are fully backwards-compatible. But it is also why I wonder if striving for that goal is a good thing.
But take a look on GTKmm? Bad library? And many others. The problem is that I am as modern library developer limited **not-to-use** boost **at-all** if I want keep ABI backward compatible.
GTKmm uses STL in its interfaces. So their ABI compatibility can only go as far as the STL ABI compatibility is maintained. If they used boost, they would stay equally compatible as long as you don't change your system boost. Best regards, Isidor
Artyom wrote:
Hello,
I want to bring up an important issue that boost developers are not aware of: anybody who develops a library and wants to keep any kind of backward compatibility can-not use Boost.
I don't think that this is correct: Sure, boost ABI changes from release to release but unless you want to provide a "one-for-all" binary release of your library, what is the issue? Even libstdc++ ABI changes every now and then. You can still provide binaries for certain distributions. For instance, Ubuntu-8.04 comes with boost-1.34 and libstdc++.so.6. That won't change. And those users which don't find a suitable binary version can still compile your library on their own, using their preferred version of boost. And they will succeed, most likely, because the API changes are pretty rare. In addition, to be honest, personally I would try to avoid a library that uses such pseudo-boost in its API instead of the real thing, because I want to use boost, not some look-alike. I don't want to use the umpteenth version of a smart pointer... Regards, Roland
Sure, boost ABI changes from release to release but unless you want to provide a "one-for-all" binary release of your library, what is the issue? Even libstdc++ ABI changes every now and then.
Not correct. GCC keeps ABI since gcc-3.4... And this is now about 6 years... Not bad?
You can still provide binaries for certain distributions. For instance, Ubuntu-8.04 comes with boost-1.34 and libstdc++.so.6. That won't change.
Yes, but debian comes with boost-1.35 and Ubuntu-8.10 would be with another version and maybe user wants to use in its application 1.42 and he can't because libfoo uses 1.35...
<snip>
I would try to avoid a library that uses such pseudo-boost in its API instead of the real thing,
As I had written I do not encourage you using this. But I can't use anything else because I can't use boost! Believe me, writing this library is far from being my first choice.
In addition, to be honest, personally because I want to use boost, not some look-alike. I don't want to use the umpteenth version of a smart pointer...
Very good point. The shared_ptr I used is boost shared_ptr but instead of inline sp_counted_base I use compiled and linked sp_counted_base so I can change implementation without breaking binary compatibility. But it is still the same shared_ptr Small difference. Artyom
Artyom wrote:
Sure, boost ABI changes from release to release but unless you want to provide a "one-for-all" binary release of your library, what is the issue? Even libstdc++ ABI changes every now and then.
Not correct. GCC keeps ABI since gcc-3.4... And this is now about 6 years... Not bad?
Were people upset back then then ? -- ___________________________________________ Joel Falcou - Assistant Professor PARALL Team - LRI - Universite Paris Sud XI Tel : (+33)1 69 15 66 35
joel falcou wrote:
Artyom wrote:
Sure, boost ABI changes from release to release but unless you want to provide a "one-for-all" binary release of your library, what is the issue? Even libstdc++ ABI changes every now and then.
Not correct. GCC keeps ABI since gcc-3.4... And this is now about 6 years... Not bad?
Were people upset back then then ?
Yes. What's your point though? It's surely impossible to hope that a given binary library will be usable for decades. But breaking ABI once in 6 years (if not more) is surely better than potentially breaking ABI 4 times per year? - Volodya
Vladimir Prus wrote:
joel falcou wrote:
Artyom wrote:
Sure, boost ABI changes from release to release but unless you want to provide a "one-for-all" binary release of your library, what is the issue? Even libstdc++ ABI changes every now and then.
Not correct. GCC keeps ABI since gcc-3.4... And this is now about 6 years... Not bad?
Were people upset back then then ?
Yes. What's your point though? It's surely impossible to hope that a given binary library will be usable for decades. But breaking ABI once in 6 years (if not more) is surely better than potentially breaking ABI 4 times per year?
- Volodya
I think that there are more variables involved which makes this optimization more complex. For instance, the dynamics of development is one of the high values of boost in my eyes. If the implementation can be improved, developers do it, which is great. ASIO is a good example. Compile time and runtime improvements in the latest release. Great. Most probably some ABI changes, too. How much improvement would justify breaking ABI? How can it be measured? Not easy, I'd say. Regards, Roland
joel falcou wrote:
Artyom wrote:
Sure, boost ABI changes from release to release but unless you want to provide a "one-for-all" binary release of your library, what is the issue? Even libstdc++ ABI changes every now and then.
Not correct. GCC keeps ABI since gcc-3.4... And this is now about 6 years... Not bad?
Were people upset back then then ?
Yes. What's your point though? It's surely impossible to hope that a given binary library will be usable for decades. But breaking ABI once in 6 years (if not more) is surely better than potentially breaking ABI 4 times per year?
But these are different matters. The gcc ABI was changed to reflect the standard ABI back then. Boost, however, claims to be a working ground to establish practises to be eventually standardized. So ABI breaking comes with the terrain and is probably even useful in order to find the best practises which deserve standardization. Best regards, Isidor
Artyom wrote:
Sure, boost ABI changes from release to release but unless you want to provide a "one-for-all" binary release of your library, what is the issue? Even libstdc++ ABI changes every now and then.
Not correct. GCC keeps ABI since gcc-3.4... And this is now about 6 years... Not bad?
My point was that all libraries change sometimes. Frequencies vary, of course. I wonder what you'll say when libstdc++ changes the next time (and it will happen)?
You can still provide binaries for certain distributions. For instance, Ubuntu-8.04 comes with boost-1.34 and libstdc++.so.6. That won't change.
Yes, but debian comes with boost-1.35 and Ubuntu-8.10 would be with another version and maybe user wants to use in its application 1.42 and he can't because libfoo uses 1.35...
Sure, that's what source distributions are for, IMHO :-)
I would try to avoid a library that uses such pseudo-boost in its API instead of the real thing,
As I had written I do not encourage you using this. But I can't use anything else because I can't use boost!
Believe me, writing this library is far from being my first choice.
OK, but as a matter of fact, I was looking for a replacement of cakephp (not sure yet), and cppcms is on the list of candidates. If using cppcms would require using booster instead of boost, I would probably strike it from the list.
In addition, to be honest, personally because I want to use boost, not some look-alike. I don't want to use the umpteenth version of a smart pointer...
Very good point. The shared_ptr I used is boost shared_ptr but instead of inline sp_counted_base I use compiled and linked sp_counted_base so I can change implementation without breaking binary compatibility. But it is still the same shared_ptr
Small difference.
OK, can I so something like the following? boost::shared_ptr<int> a(new int(0)); booster::shared_ptr<int> b = a; (replace shared_ptr with all the other boost-look-alike classes) Regards, Roland
OK, but as a matter of fact, I was looking for a replacement of cakephp (not sure yet), and cppcms is on the list of candidates. If using cppcms would require using booster instead of boost, I would probably strike it from the list.
No, you would not be required to use Booster anywhere. But if you want to interact with cppcms library in very specific points. For example you want to get shared_ptr<cppcms::http::context> or you want to use cppcms::widgets::regex_field then you would have to use booster. Also the largest exposed part of booster is boost.locale that is actually the same code in different namespace. There are very few points that actually require such glue. But sometimes they are must-to-have.
OK, can I so something like the following?
boost::shared_ptr<int> a(new int(0)); booster::shared_ptr<int> b = a;
(replace shared_ptr with all the other boost-look-alike classes)
No you can't assign boost::shared_ptr to booster::shared_ptr as you can't assign std::tr1::shared_ptr to boost::shared_ptr But you can freely replace one with other as long as they consistent. Best, Artyom
On Sat, May 15, 2010 at 10:53 AM, Artyom <artyomtnk@yahoo.com> wrote:
OK, can I so something like the following?
boost::shared_ptr<int> a(new int(0)); booster::shared_ptr<int> b = a;
(replace shared_ptr with all the other boost-look-alike classes)
No you can't assign boost::shared_ptr to booster::shared_ptr as you can't assign std::tr1::shared_ptr to boost::shared_ptr
But you can freely replace one with other as long as they consistent.
But that could then ripple through an entire codebase - 100Ks, most of it not mine to change. What might, instead, be possible is to make some bridge classes. library.h: inline boost::shared_ptr<int> some_api() { boost::abi::shared_ptr<int> asp = some_api_abi(); // call the real function return boost::shared_ptr<int>(asp, boost::abi::deleter<int>); } ***Or something like that*** Importantly, note that since this is in your h file, which I include, I will get *my* version of shared_ptr returned from some_api(). It will be a wrapper around your shared_ptr, thus with extra overhead, but at least it will work. Basically your functions return a abi version of shared_ptr, which may need to be pure interface, if you really want to be sure about abi: template <typename T> struct boost::abi::shared_ptr { virtual void delete() = 0; virtual void add_ref() = 0; //... }; Or maybe it is something else, but it is a simple bridge that you can hopefully keep compatible for a number of years, and can be converted to *my* shared_ptr. Lather, rinse, repeat for other ABIs. Yes, this may be cumbersome, but no more than what you are already doing. Tony
template <typename T> struct boost::abi::shared_ptr { virtual void delete() = 0; virtual void add_ref() = 0; //... };
Or maybe it is something else, but it is a simple bridge that you can hopefully keep compatible for a number of years, and can be converted to *my* shared_ptr.
Lather, rinse, repeat for other ABIs. Yes, this may be cumbersome, but no more than what you are already doing.
It is probably comparable to what boost::serialization does in terms of stripping of implementation details, which is why I wondered if a boost::abi framework could become a new boost module at some time. Best regards, Isidor
Hello Boosters, Some clarifications: 1. I do not suggest the way boost works but I do suggest a community to think about something like boost stable. Today it is quite feasible to take all tr1 stuff and make it ABI stable sacrificing some generality. This would give very good to C++. 2. Booster is not a fork of boost but rather small library that allows me to get free of boost keeping boost-like API. There are many discussions about Boost issues today (i.e. "is boost broken?") - but I think there is important point that is missed - don't forget about it. Artyom
Hello Boosters,
Some clarifications:
1. I do not suggest the way boost works but I do suggest a community to think about something like boost stable.
I am just doubting that boost is the right place to tackle the ABI stability issue. Much like I think the build/test system development is better split out of boost and acquired through CMake, I think ABI stabilizing also belongs somewhere else: in distributors' responsibility. What I would rather like to see in boost would be building blocks which go better with the dynamic nature of boost development, but which could become tools for distributors putting together a stable boost. These could be: (a) Granular ABI versioning for boost components: Imagine being able to put a... int shared_ptr_abi() { return boost::shared_ptr<int>::abi_version; } ..in your shared library, and then checking for compatibility using: int main() { assert(boost::shared_ptr<int>::abi_version == shared_ptr_abi()); ... So there are no subtle bugs through ABI changes, but the program just refuses to work, notifying you about recompiling both components against the same boost version. (b) modularization of boost components: Suppose I have many libraries using boost::shared_ptr in their exported interface, while boost::asio is used extensively, but none of its objects are exposed in interfaces. Then I would prefer being able to upgrade just boost::asio, which would allow me to upgrade libraries depending on it one by one, and leave the boost::shared_ptr upgrade out until it becomes feasible to upgrade all the libraries at once.
Today it is quite feasible to take all tr1 stuff and make it ABI stable sacrificing some generality.
This would give very good to C++.
2. Booster is not a fork of boost but rather small library that allows me to get free of boost keeping boost-like API.
Maybe "fork" was bad wording considering that the code is not derived from boost. Still, as it serves as a boost replacement, I am saying that I would love to see improvements you put there to be put into boost instead, so they serve a larger audience. Best regards, Isidor
Maybe "fork" was bad wording considering that the code is not derived from boost. Still, as it serves as a boost replacement, I am saying that I would love to see improvements you put there to be put into boost instead, so they serve a larger audience.
This is just a question of priorities: I want to release CppCMS - I need stable boost-like library. 1. I can't fix all boost on-my-own. 2. I can extract small parts and implement exacly what I need. So I do this. So I do have my own reasons, and BTW Boost.Locale is written with backward ABI compatibility in mind. So once boost "stabilization" would be started as official part of boost I'll be the first who helps. Artyom
Maybe "fork" was bad wording considering that the code is not derived from boost. Still, as it serves as a boost replacement, I am saying that I would love to see improvements you put there to be put into boost instead, so they serve a larger audience.
This is just a question of priorities: I want to release CppCMS - I need stable boost-like library.
Simple: 1. You release the CppCMS source (AFAIR, it is meant to be open source), people download the source, compile it, compile their applications (both against their installed boost). So it will work as they will have the same boost ABI in both components. 2. (optional) You release binaries compiled for the distributions you use. They will be compiled against the boost version shipped with that distributions, so people running the same distribution can use them and avoid compiling themselves. 3. (once CppCMS gains popularity) People running other distributions release unofficion binaries compiled for their distributions. The same as in (2) applies. 4. (once CppCMS gains even more popularity) Distributors ship official compiled builds of CppCMS for their distributions. The chances for (3) and (4) might be significantly reduced once CppCMS depends on an extra library which implements functionality that could instead be obtained from boost with much more peer review improving it.
1. I can't fix all boost on-my-own.
Me too. So if I find a bug in boost, I publish it. Others do the same. The effect is a boost with very high code quality (I think it happened only once for me that deploying software was slightly delayed due to a boost bug).
So once boost "stabilization" would be started as official part of boost I'll be the first who helps.
Look for a distributor who maintains ABI compatibility and send them you resume. If they lack someone who is expert in boost ABI engineering and you can fill that need, they will probably hire you. Best regards, Isidor
So I do have my own reasons, and BTW Boost.Locale is written with backward ABI compatibility in mind.
So once boost "stabilization" would be started as official part of boost I'll be the first who helps.
Don't get me wrong, though. I do find it valuable to explore solutions to solve ABI problems. I just would not like to see the boost community being split into those who work on what boost is now, and those who are always caught in a "should we fix this bug? it makes applications break, but fixing it would break the ABI" dilemma. I think a particular virtue of boost is that bugs are fixed, and that they are fixed in the exact place where the fix belongs, even if that involves breaking the ABI. Everything else would be a workaround to some extent. But I do find it useful to have some control over the ABI. Not because I want to deploy software in environments where people randomly throw together modules compiled against different boost versions, but because I need updating strategies for large deployments of binary modules which are a bit more granular than "upgrade boost, then recompile everything". However, today's possibilities are different from what was possible before. I think fixed binary ABIs are less important today than they were some years ago. Today, good compilers are affordable, computing power sufficient for compiling is affordable, sophisticated source code management systems are available, programming paradigms have evolved etc. This allows for more heterogenous environments. We do not need to strive for having all software converge to a fixed goal which doesn't evolve anymore (just waiting for the day it breaks), but we can have software evolve even more dynamically because there are tools to make sure that all components still find other components they are compatible to. Best regards, Isidor
those who are always caught in a "should we fix this bug? it makes applications break, but fixing it would break the ABI" dilemma.
ABI compatibility should not prevent fixing bugs. This is important point. In fact libstdc++ manages its ABI for 6 years successfully, Qt does this very well too. However you need to be prepared. You generally need at least one d-pointer/opaque-pointer/pimpl. You need to separate implementation and definitions (it would also increase compilation time, and reduce code size). Only bugs you can't fix are deep design issues. I don't see it as problem. Take a look on Boost.Locale source code I had written. Almost all non-trivial code put into C++ source files. Every non-trivial class has pimpl pointer allowing me to make changes quite easily and so on.
I need updating strategies for large deployments of binary modules which are a bit more granular than "upgrade boost, then recompile everything".
Exactly, this one of very important issues that ABI stability would solve. Artyom
On 16 May 2010, at 08:00, Artyom wrote:
those who are always caught in a "should we fix this bug? it makes applications break, but fixing it would break the ABI" dilemma.
ABI compatibility should not prevent fixing bugs. This is important point. In fact libstdc++ manages its ABI for 6 years successfully, Qt does this very well too.
However you need to be prepared. You generally need at least one d-pointer/opaque-pointer/pimpl. You need to separate implementation and definitions (it would also increase compilation time, and reduce code size).
libstdc++ does none of these things. Instead what libstdc++ does is implement a fixed interface (as specified by the standard), and accepts the code is "ABI frozen", accepting the pain this can bring because ABI stability is important. Chris
libstdc++ does none of these things. Instead what libstdc++ does is implement a fixed interface (as specified by the standard), and accepts the code is "ABI frozen", accepting the pain this can bring because ABI stability is important.
Not exactly. All pure template classes like STL containers do not have pimpl for obvious reasons but... - std::locale class is fully pimpled. - All standard locale facets are fully pimpled and most complicated operations moved into implementation and are subject to change. - If I'm not mistaken file/io is pimpled as well. So complex stuff like locale, facets is hidden. Artyom
those who are always caught in a "should we fix this bug? it makes applications break, but fixing it would break the ABI" dilemma.
ABI compatibility should not prevent fixing bugs. This is important point. In fact libstdc++ manages its ABI for 6 years successfully, Qt does this very well too.
These are very different cases. libstdc++ implements functionality which has been standardized for a long time. Qt uses a lot more indirection than boost does.
However you need to be prepared. You generally need at least one d-pointer/opaque-pointer/pimpl. You need to separate implementation and definitions (it would also increase compilation time, and reduce code size).
Honestly, if boost was changed to use these indirection techniques in all places, I'd probably fork it or go for an alternative solution. Indirection might solve the fragile interface problem and reduce compilation times and code size, but it also effectively blocks any compiler optimization which spans multiple methods that could be inlined before. Consider boost::phoenix: as we have it now, without indirection, I can write expressions using phoenix and count on a decent compiler to optimize all the classes out, leaving me with native code which often is no different than if I had coded the expressions by hand. A pimpled boost::phoenix would result in lots of indirect calls all the time, which would badly hurt performance. As stated before, I would consider it useful to have a boost::abi module which abstracts away the internals. This would probably use one of the indirection techniques you mentioned. But I don't think something like this should be scattered throughout the whole boost. I would not want to have yet another lightweight layer which could fit between boost and C++. I'd rather think of boost as a toolset sitting right on the top of C++, which is not necessarily binary stable, but can be used to build the hard bits of a binary stable library. BTW, I would rather compare boost not with Qt or libstdc++, but with Linux kernel development. They also work at a very low level, and they came to the conclusion that a fixed ABI would be to costly. I think a fixed ABI is something that should only be considered on a higher level.
I need updating strategies for large deployments of binary modules which are a bit more granular than "upgrade boost, then recompile everything".
Exactly, this one of very important issues that ABI stability would solve.
It surely would. But I do not regard it as the best solution. A stable ABI would just mean even larger deployments using the same ABI, just waiting for the day where some design issue makes it unavoidable to break the ABI. Then the problem is even bigger. A better solution, at least in the boost case, would be to make a volatile ABI more manageable. Best regards, Isidor
Honestly, if boost was changed to use these indirection techniques in all places, I'd probably fork it or go for an alternative solution.
I do not suggest having all boost abi-stable but take most valuable stable and important parts (tr1 for example) and make them such or, add ::boost::stable or ::unboosted namespace and put there all stable parts.
Indirection might solve the fragile interface problem and reduce compilation times and code size, but it also effectively blocks any compiler optimization which spans multiple methods that could be inlined before. Consider boost::phoenix: as we have it now, without indirection,
Of course not every part should be pimpl-ed and many can't.
As stated before, I would consider it useful to have a boost::abi module which abstracts away the internals. This would probably use one of the indirection techniques you mentioned. But I don't think something like this should be scattered throughout the whole boost.
Agree
BTW, I would rather compare boost not with Qt or libstdc++, but with Linux kernel development. They also work at a very low level, and they came to the conclusion that a fixed ABI would be to costly. I think a fixed ABI is something that should only be considered on a higher level.
This is good point. Linux kernel unstable for Driver development so you need to compile each driver with specific kernel (or just get it into kernel). But it is SUPER stable for user. System calls are never (or almost) never removed and keep their compatibility. So given right libraries you can run code compiled for Linux 2.0 on Linux 2.6.32 Artyom
So I do have my own reasons, and BTW Boost.Locale is written with backward ABI compatibility in mind.
So once boost "stabilization" would be started as official part of boost I'll be the first who helps.
But why not make a boost trunk clone on gitorious.org and start a boost stable branch yourself? It would probably need some documented update policy (like: sync down from trunk every maybe 2 or 5 years, and in between only ABI-neutral changes are imported and bugs fixed) so users will know they will not be caught in a fork, but have a defined schedule on when to expect new features from mainline boost coming in. But then I could imagine interesting contributors or at least people providing feedback coming along themselves. Best regards, Isidor
But why not make a boost trunk clone on gitorious.org and start a boost stable branch yourself? It would probably need some documented update policy (like: sync down from trunk every maybe 2 or 5 years, and in between only ABI-neutral changes are imported and bugs fixed) so users will know they will not be caught in a fork, but have a defined schedule on when to expect new features from mainline boost coming in. But then I could imagine interesting contributors or at least people providing feedback coming along themselves.
There is three major issues with this approach: 1. It is fine as long as I use one working version of boost and it has no bugs. Because once I need bug fixes, it would be non-trivial to fix them in the state boost today is. Without correct code preparation it maybe impossible to fix bugs without breaking ABI. So it is no-go solution. 2. Some boost ABI code may be affected by defines. For example if user defines debug smart pointer option just to see how it works in its own code it may crash the application because library sees somethig else. 3. Boost is configured mostly with defines. So upgrade of compiler version for gcc-4.3 to gcc-4.4 may break ABI just because some new feature comes and something is changed in class layout. So this not simple thing to do. Example: In Boost I changed shared_ptr a little to prevent these issues: 1. I removed debug hooks from shared/weak_ptr's 2. I moved sp_counted_base code to implementation and added mutex instantly to it (if I even do not use it). So I can change its behavior without affecting the rest of the code. Artyom
But why not make a boost trunk clone on gitorious.org and start a boost stable branch yourself? It would probably need some
..
3. Boost is configured mostly with defines. So upgrade of compiler version for gcc-4.3 to gcc-4.4 may break ABI just because some new feature comes and something is changed in class layout.
Ok, I better understand your approach now. But I also have to say that this shows yet again how a boost::stable design may be fundamentally different from what boost does: one great use is to abstract away many compiler differences. If boost was to be binary stable and still serve this purpose, this would mean that for one buggy compiler boost supports, all compilers would have to pay the price because the ABI must be designed in a way to be able to incorporate the particular workaround. Best regards, Isidor
Artyom wrote:
OK, but as a matter of fact, I was looking for a replacement of cakephp (not sure yet), and cppcms is on the list of candidates. If using cppcms would require using booster instead of boost, I would probably strike it from the list.
No, you would not be required to use Booster anywhere. But if you want to interact with cppcms library in very specific points.
For example you want to get shared_ptr<cppcms::http::context> or you want to use cppcms::widgets::regex_field then you would have to use booster.
Also the largest exposed part of booster is boost.locale that is actually the same code in different namespace.
There are very few points that actually require such glue. But sometimes they are must-to-have.
Well, instead of writing glue code and wondering (for example) how to encapsulate the usage of booster::shared_ptr or pondering over ownership questions/issues if I transfer the pointer to a boost::shared_ptr, I'd prefer cppcms API to use boost-original. I'd compile the library myself, anyway, which relieves me of the ABI troubles. Regards, Roland
Well, instead of writing glue code and wondering (for example) how to encapsulate the usage of booster::shared_ptr or pondering over ownership questions/issues if I transfer the pointer to a boost::shared_ptr, I'd prefer cppcms API to use boost-original.
I'd compile the library myself, anyway, which relieves me of the ABI troubles.
My current stable version of CppCMS uses boost in its API and relays on boost-signals, boost-iostreams, boost-system and boost-regex. Have you tryed to deploy such library to remote host? I think you hadn't? Even I'm myself, had huge troubles deploying hotfix to production server that was running cppcms because the production server had different version of boost. And both servers were under my total control... Such dependencies are very critical. Especially for CppCMS uses that have great headaches deploying their applications to other environments. For next version it would be much simpler. As it today simple for KDE, Gnome, GTK or Qt based applications. Artyom
Artyom wrote:
Well, instead of writing glue code and wondering (for example) how to encapsulate the usage of booster::shared_ptr or pondering over ownership questions/issues if I transfer the pointer to a boost::shared_ptr, I'd prefer cppcms API to use boost-original.
I'd compile the library myself, anyway, which relieves me of the ABI troubles.
My current stable version of CppCMS uses boost in its API and relays on boost-signals, boost-iostreams, boost-system and boost-regex.
Have you tryed to deploy such library to remote host? I think you hadn't?
I do that almost every Wednesday around noon with about 100 libraries, not kidding :-)
Even I'm myself, had huge troubles deploying hotfix to production server that was running cppcms because the production server had different version of boost. And both servers were under my total control...
Boost is part of the deployment. Thus, I can always rely on having the right version. Regards, Roland
2010/5/15 Artyom <artyomtnk@yahoo.com>:
Have you tryed to deploy such library to remote host? I think you hadn't?
Even I'm myself, had huge troubles deploying hotfix to production server that was running cppcms because the production server had different version of boost. And both servers were under my total control...
Hi, To make this easier, I make my applications an external-process plugins and compile with static Boost libraries. Then I can deploy several applications using different versions of Boost to the same server, which is not controlled by me and may have no Boost installed at all. Alexander Churanov
2010/5/15 Artyom <artyomtnk@yahoo.com>:
Have you tryed to deploy such library to remote host? I think you hadn't?
Even I'm myself, had huge troubles deploying hotfix to production server that was running cppcms because the production server had different version of boost. And both servers were under my total control...
Hi,
To make this easier, I make my applications an external-process plugins and compile with static Boost libraries. Then I can deploy several applications using different versions of Boost to the same server, which is not controlled by me and may have no Boost installed at all.
Doesn't reduce that approach portability, at least for the external-process communication part? Best regards, Isidor
You can still provide binaries for certain distributions. For instance, Ubuntu-8.04 comes with boost-1.34 and libstdc++.so.6. That won't change.
I think this is the best direction for solving the ABI compatibility issue. Why should a library vendor bother? The library vendor would sacrifice code quality if he strives for binary compatibility. A distributor, however, can decide to make binary compatibility priority, and keep the same boost version for a long time. So, if I need long-lasting binary compatibility, I can buy RHEL and be sure everything remains stable. But the developers of all the bundled software don't need to bother. Best regards, Isidor
On 15 May 2010, at 18:40, Isidor Zeuner wrote:
You can still provide binaries for certain distributions. For instance, Ubuntu-8.04 comes with boost-1.34 and libstdc++.so.6. That won't change.
I think this is the best direction for solving the ABI compatibility issue. Why should a library vendor bother? The library vendor would sacrifice code quality if he strives for binary compatibility. A distributor, however, can decide to make binary compatibility priority, and keep the same boost version for a long time.
So, if I need long-lasting binary compatibility, I can buy RHEL and be sure everything remains stable. But the developers of all the bundled software don't need to bother.
There is one big difference between libstdc++ and boost. libstdc++ fixes bugs in old versions, while keeping the ABI fixed. boost makes almost no attempt to apply bug fixes to old versions. I'm not claiming it should, but saying that a distributor can deal with keeping everything stable is unreasonable, unless you expect RHEL to keep their own boost distribution, where they backport bug fixes. Chris
On 15 May 2010, at 18:40, Isidor Zeuner wrote:
You can still provide binaries for certain distributions. For instance, Ubuntu-8.04 comes with boost-1.34 and libstdc++.so.6. That won't change.
I think this is the best direction for solving the ABI compatibility issue. Why should a library vendor bother? The library vendor would sacrifice code quality if he strives for binary compatibility. A distributor, however, can decide to make binary compatibility priority, and keep the same boost version for a long time.
So, if I need long-lasting binary compatibility, I can buy RHEL and be sure everything remains stable. But the developers of all the bundled software don't need to bother.
There is one big difference between libstdc++ and boost.
libstdc++ fixes bugs in old versions, while keeping the ABI fixed. boost makes almost no attempt to apply bug fixes to old versions.
libstdc++ has to, because it is bound to implementing well-defined requirements. Boost wants to explore how to get the most out of C++, which naturally goes with a more progressive strategy. Once boost components get standardized, compiler vendors will run for them to carve them out of stone anyway.
I'm not claiming it should, but saying that a distributor can deal with keeping everything stable is unreasonable, unless you expect RHEL to keep their own boost distribution, where they backport bug fixes.
Well, this backport-instead-of-upgrade policy is pretty much the reason why people buy RHEL. Almost 5 years of boost-1.33.1 don't come without a reason. Best regards, Isidor
Artyom wrote:
Hello,
I want to bring up an important issue that boost developers are not aware of: anybody who develops a library and wants to keep any kind of backward compatibility can-not use Boost.
it is impossible to promise that it would not be broken between releases.
Why boost::details::sp_counted_base may be changed because in next version atomic counter was implemented using assembly rather then pthread's mutex... which affects the layout of boost::details::sp_counted_base and thus breaks compatibility.
I've heard this many times and I have to say I'm not getting it. case I First let's examine a library implemented as header only. a) version 0 - a bunch of headers. User builds his code and is pleased as punch. b) version 1 - same bunch of headers + plus a few more. Also the original classes have some new functions - BUT none of the existing functions are eliminated nor have a change in interface. Where is the incompatibility. What does "binary compatibility" even mean since the the application doesn't use the new functionality until it is rebuil? But this can't be a problem. To my knowledge - I don't think that any libraries deprecate existing functions nor change thier interface or sematics. So I don't understand where this whole thing is coming from case II A library that has to be built as either a DLL or a static library. a) version 0 - bunch of headers and source modules. User is pleased as punch same as above. b) version 1 - same as above + changes in function implementation Now there a library/dll which is different than the one produced by version 0. User rebuilds his app against the new library. It's still going to work since the newer version has everything the old version does + some more. Also SUPPOSE that no data members are deleted/added to the header declarations corresponding to interface classes. c) User rebuilds his app and sends it to his customers how have the DLL library built in version 0. if he's using a static library there's no problem. If he's using a DLL the application will run against the original version DLL with no problem. Sooo it seems to me that the only situation which would "break binary compatibility" would be when ALL the following are true. i) The library is not a header only library ii) class definition includes new data members in one or more interface. iii) one library is distributed separately from it's prequisites Am I getting this right? If so it would seem that there's not much that "breaks binary compatibility" and that it wouldn't take too much effort to avoid this. What am I missing here? Robert Ramey
First let's examine a library implemented as header only.
a) version 0 - a bunch of headers.
User builds his code and is pleased as punch.
b) version 1 - same bunch of headers + plus a few more. Also the original classes have some new functions - BUT none of the existing functions are eliminated nor have a change in interface.
Where is the incompatibility.
[snip]
In this case two library are compatible unless: - New introduced functions are virtual and user derives its class - The class has new members - so layout of the class has changed. Example. boost->v1-50 shared_ptr uses sp_counted_base that has following members use_count_ weak_count_ mutex_ boost->v1->51 shared_ptr implemented atomic ops for this platform and sp_counted_base has following members: use_count_ weak_count_ mutex_ ABI broken since in one part it uses atomic ops and not locks mutex in other it uses only mutex - code is not thread safe.
case II
A library that has to be built as either a DLL or a static library.
a) version 0 - bunch of headers and source modules.
User is pleased as punch same as above.
b) version 1 - same as above + changes in function implementation Now there a library/dll which is different than the one produced by version 0.
User rebuilds his app against the new library.
Note, once user rebuilds the library against new library everithing is fine as long as API compatible (and boost keeps the API **mosty** compatible. This is not the issue.
c) User rebuilds his app and sends it to his customers how have the DLL library built in version 0. if he's using a static library there's no problem. If he's using a DLL the application will run against the original version DLL with no problem. [snip]
Same, rebuild solves the issue.
Sooo it seems to me that the only situation which would "break binary compatibility" would be when ALL the following are true.
i) The library is not a header only library ii) class definition includes new data members in one or more interface. iii) one library is distributed separately from it's prequisites
Am I getting this right? If so it would seem that there's not much that "breaks binary compatibility" and that it wouldn't take too much effort to avoid this.
No, it is much more complicated. There actually two major cases, that should be solved. (a) User has a library libfoo.(so|dll|lib|a) that was compiled against header only or linked (statically or dynamically) boost-1.33.1 Now, customer what to use libfoo but he needs boost-1.35.0 He can't because 1.35 is not ABI compatible to 1.33.1 So any class that passes between libfoo and customer app that uses boost would have danger of crash. More then that, under ELF platforms (Linux, Solaris, BSD) there is even bigger issue because this may happen even if you do not share classes explicitly between libfoo and your app. Because all symbols are uniquely mapped so user of libfoo may get boost-1.35 symbols even if they are used internally. (b) User developed its application against boost-1.33.1 dll/so, now after system upgrade new version of boost installed now its application would not start because no 1.33 available. This is little bit simple case because it just need to keep two version of boost dll 1.33.1 and 1.35. The first issue is my biggest problem using boost-as-is. Now how ABI can be broken: -------------------------- - sizeof of class has changed - new virtual function was introduced - the semantics of private members had changed and they are used from header only libraries. - and much more This is great tutorial about it: http://techbase.kde.org/Policies/Binary_Compatibility_Issues_With_C%2B%2B Now, you will see that keeping ABI is quite tricky. I hope that this explains the issue. Artyom
More then that, under ELF platforms (Linux, Solaris, BSD) there is even bigger issue because this may happen even if you do not share classes explicitly between libfoo and your app. Because all symbols are uniquely mapped so user of libfoo may get boost-1.35 symbols even if they are used internally.
Now this is an interesting issue. We stumbled over the same issue when discussing how to best integrate boost in CLucene ([1]). I think boost could use a user-controlled name mangling facility so a binary library distributor could make sure the symbols from the boost he ships do not collide with boost symbols in other code. I haven't tried if a -Dboost=boost_1_43 preprocessor flag (applied to both the boost build and the application build) is sufficient, though. Best regards, Isidor [1] http://clucene.sourceforge.net/
Now this is an interesting issue. We stumbled over the same issue when discussing how to best integrate boost in CLucene ([1]). I think boost could use a user-controlled name mangling facility so a binary library distributor could make sure the symbols from the boost he ships do not collide with boost symbols in other code.
There are few ways to do it. If you have header-only library you may compile all your source code with visibility=hidden gcc option but you need to manage what do you export similarly as it is done in windows marking classes with default visibility. However it is quite tricky and this would not work with compiled libraries.
I haven't tried if a -Dboost=boost_1_43 preprocessor flag (applied to both the boost build and the application build) is sufficient, though.
There is my small python script that renames boost [1]. It renames all includes and defines so you can simultaneously use several boost version in same code. Also latest BCP version allow you to do similar job, but it does not rename includes and macros so you can't include two boost version in same C++ code. Note: my script was written before BCP provided such option. **BUT!!!!** It does not solves all problems: 1. There is a code placed in tr1 namespace - it would collide if you use it. 2. Some libraries that provide extern "C" functions do bad job and pollute global namespace with these symbols. They collide as well. Examples: - Boost.Thread: https://svn.boost.org/trac/boost/ticket/3812 - Boost.Test: https://svn.boost.org/trac/boost/ticket/3811 - LW Thread: https://svn.boost.org/trac/boost/ticket/3810 But in some libraries this behavior is actually design. For example boost.regex provides POSIX regex API regcomp(A|W), regexec(A|W) that go directly to global namespace. (BTW kill me if understand who created this A|W api - POSIX does not have this nonsense). It looks like that Boost C++ developers are not really familiar with correct methods of providing replacement API in C. So you have to compile boost version without this API manually... As you can see if I want to use boost.thread even namespace renaming would not help me till these bugs are fixed. I hope this helps, Artyom P.S.: as you can see it is quite non-trivial issue. [1]: http://cppcms.svn.sourceforge.net/viewvc/cppcms/framework/branches/refactoring/tools/rename.py?revision=1128&view=markup
I haven't tried if a -Dboost=boost_1_43 preprocessor flag (applied to both the boost build and the application build) is sufficient, though.
There is my small python script that renames boost [1]. It renames all includes and defines so you can simultaneously use several boost version in same code.
Thanks for the pointers. This already sounds very useful for bundling boost. Actually, I'd think it would also solve your problem: you could bundle a renamed boost and control yourself what updates you allow. Thereby you allow a binary stable build. Yet it would be still possible to provide a compile time flag to use the system boost instead. So people comfortable with a changing ABI can avoid having two boosts, and people uncomfortable with it don't need to have that issue. But what I would like even more to see would be boost solving the renaming problem without copying, by just using the preprocessor. Think replacing all the namespace boost { ... } with something like: BOOST_NS_START .. BOOST_NS_END ..and boost::foo with BOOST_NS::foo. Then the BOOST_NS* macros could allow to have preprocessor-controlled namespace naming.
Also latest BCP version allow you to do similar job, but it does not rename includes and macros so you can't include two boost version in same C++ code.
I do not see a point in using two boost versions in the same compilation unit. This would just increase code size.
Note: my script was written before BCP provided such option.
**BUT!!!!** It does not solves all problems:
1. There is a code placed in tr1 namespace - it would collide if you use it. 2. Some libraries that provide extern "C" functions do bad job and pollute global namespace with these symbols. They collide as well.
Examples: - Boost.Thread: https://svn.boost.org/trac/boost/ticket/3812 - Boost.Test: https://svn.boost.org/trac/boost/ticket/3811 - LW Thread: https://svn.boost.org/trac/boost/ticket/3810
But in some libraries this behavior is actually design. For example boost.regex provides POSIX regex API regcomp(A|W), regexec(A|W) that go directly to global namespace.
Thanks for the pointers. I think it would be a great benefit for boost to make all these global names preprocessor-configurable.
P.S.: as you can see it is quite non-trivial issue.
I already feared it was that way :). But I think this is the way to go. Maybe we can have multiple-parallel-boost-versions regression tests some day. But this would call for a distributed SCM system like git for boost development, so an automated test could pull an older (incompatible) version to compile against without bothering the svn server all the time. Best regards, Isidor
I do not see a point in using two boost versions in the same compilation unit. This would just increase code size.
It helps if libfoo uses boost-1.33.1 in its interface and you must use boost-1.43 for some feature that boost-1.33.1 do not have so you can actually use both versions in same unit. Of course it is not best practice but this is an issue that should be considered. Also if you do use two version of boost in same application it would be much easier to distinguish between them because you can include <my_boost/foo.h> and <your_boost/bee.h> Think of that it is possible, especially when you have 3rd part library.
Thanks for the pointers. I think it would be a great benefit for boost to make all these global names preprocessor-configurable.
Agree, but till that comes bcp would be very helpful. Artyom
Isidor Zeuner wrote:
More then that, under ELF platforms (Linux, Solaris, BSD) there is even bigger issue because this may happen even if you do not share classes explicitly between libfoo and your app. Because all symbols are uniquely mapped so user of libfoo may get boost-1.35 symbols even if they are used internally.
Now this is an interesting issue. We stumbled over the same issue when discussing how to best integrate boost in CLucene ([1]). I think boost could use a user-controlled name mangling facility so a binary library distributor could make sure the symbols from the boost he ships do not collide with boost symbols in other code.
We manage all of this stuff quite easily. We have multiple, staged Boost releases on our servers (auto mounted on *nix and in subdirectories of a fixed location on a Windows server mapped to a fixed drive letter on the client machines). Each application or shared library (DSO or DLL) has runtime path information compiled in, so it knows the versions of the shared libraries it needs. The various releases of our in-house shared libraries are also staged on the servers. Thus, various applications can run on the same box, dynamically linking to whichever libraries it was built against. The assumption that a library client can, or should be able to, use whatever Boost build they happen to have lying about with their code while using a third party library built with another version of Boost is the problem. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
The assumption that a library client can, or should be able to, use whatever Boost build they happen to have lying about with their code while using a third party library built with another version of Boost is the problem.
This is exactly what today's build/autoconfiguration schemes will do in most cases. If the library client is configured to depend on libfoo (compiled against a boost other than the system boost) and boost, autoconfiguration will normally cause symbols from both the system boost and libfoo's boost to collide in the resulting code. As boost is by design likely to create ABI incompatible symbols with different versions, a robust, user-controllable name mangling facility would be useful. Best regards, Isidor
"Artyom" <artyomtnk@yahoo.com> wrote in message news:982320.15795.qm@web36701.mail.mud.yahoo.com... Hello,
That is not a first time I bring up this issue.
Exactly, and I must say I am still puzzled by the line of some of your arguments so far... For example on the one hand you will complain about the design of Boost in general and that Boost.ASIO produces code that is too big/bloated, while OTOH you will praise Qt and claim things like 'virtual functions and pimpls have negligable efficiency impact'... For the simple sake of consistency you simply cannot claim _both_ of these 'sides'...: - Boost is praised precisely for its design by leading figures of the 'C++ world' and it has acted as the incubator for functionality that eventualy got included into the standard library itself...OTOH Qt's design and philosophy can for the most part be described as downright horrible, it is not exception safe ('lives' in some fantasy single-process infinite memory world), sometimes does not even assert that an OS API call succeeds (much less error-handle it), uses non standard tools and macros, has unclear ownership semantics (you still new everything and pass around raw pointers hoping you got it right who or what will eventually delete it, they have just relatively recently started to learn the basics of RAII, and of course went and reinvented the umpteenth smart pointer), if I rember correctly even in verion 4.5 I could still find places where they checked for null return from new....... - the 'negligable efficienty impact' claim is just plain wrong as proven exactly by Qt...it is quite more than enough to investigate the horrifying amount of code that gets executed and memory chunks that get allocated for each and every window message that your application receives even if no widget wants it or handles it (it finally gets 'eaten' by an empty virtual function in some base class) _precisely_ because of the injudicious (ab)use of virtual functions and pimpls (along with other such 'wonderful and negligable' idioms that I like to call optimizer brickwalls).....in other words praising a library like Qt whose sole core GUI functionality compiles to sizes like ~10MB (!? 'sheesh' I mean it's only supposed to wrap existing OS functionality) and at the same time complaining about the compiled size of Boost.ASIO just fails the consistency test... Furthermore, even the libraries you mentioned, like Qt, do not provide 'infinite' backward compatiblity...i.e. Qt 5 will break the ABI compatiblity with Qt 4...So you are pretty much forced to the updates anyway...the issue is really only of the time interval...And since we are dealing with C++ here, your much bigger problem is the nonexisting ABI guarantees of the language itself...in the general case, if you upgrade or change your compiler bye bye ABI compatiblity...if you really want a stable ABI you'd have to go for a C one... As has already been suggested, why don't you just simply link statically with Boost? One of the many good sides of efficiently and tightly written libraries is that you can just simply link statically (not that I am claiming that all of Boost fits into that category, unfortunately it does not...but compared to Qt......)... You must realize that what you are asking from Boost is simply not feasible and seems reasonable only to you or a subset of Boost users...For example, if I understood you correctly, you would even like to have shared_ptr<> pimpled...while for example to me personally this idea is 'alarming' if not 'criminal' because I don't like shared_ptr<> various overheads even as it is...I doubt that even Qt will eventually pimpl its variant of shared_ptr... In other words...if you dislike the Boost 'philosophy' so much, why do you even use it..? ...most of its functionality can be found in other libraries (like ACE instead of ASIO...)... -- "What Huxley teaches is that in the age of advanced technology, spiritual devastation is more likely to come from an enemy with a smiling face than from one whose countenance exudes suspicion and hate." Neil Postman
Hello Domagoj Saric Before I start answering each particular point I want to say in general: 1. Boost is great library, it is one of the best things that happened to C++. 2. Boost is rather evolutes that follows strict design, this is good but it is also not so good sometimes. 3. Boost is not free of many deep problem, as an example - total lack of maintenance releases - upgrade or stay with bugs. But instead of protecting Boost as "holy" library I suggest - face existing problems and deal with them and accept that - Boost is not bug free (and it isn't) - Boost has some design and maintenance issues. - Boost does not fit anywhere. Now lets go to each particular
For example on the one hand you will complain about the design of Boost in general and that Boost.ASIO produces code that is too big/bloated, while OTOH you will praise Qt and claim things like 'virtual functions and pimpls have negligable efficiency impact'... For the simple sake of consistency you simply cannot claim _both_ of these 'sides'...:
Not right. And I explain why. Heavy template code produces significantly lager code. So hiding implementation actually reduces code size and reduces bloat. Sometimes I just prefer to move some members to pimpl just in order to prevent inclusion of 10 more useless headers. Good example is Linux kernel. As policy it does not accept macros and inline functions in code. (Similar to templates in C++ or in-header implementations) because it reduces code size and thus significantly reduces cache misses. If Linux was written for Modern C++ compiler with "Modern" C++ design in Boost style, it would probably would not be able to run on computers with several MB of memory.
OTOH Qt's design and philosophy can [snip[ has unclear ownership semantics
In Qt, you have quite strict policy of ownership and if you play by Qt rules you are safe. Just to remind Qt exists from days where C++ was hopeless kid and compilers where awful. It developed its own methods and policies and today is still works very nice with modern design.
(you still new everything and pass around raw pointers hoping you got it [snip] whose sole core GUI functionality compiles to sizes like ~10MB (!? 'sheesh' I mean it's only supposed to wrap existing OS functionality)
Just for the record. Nobody says that Qt is problem free, but today it is one of the best GUI libraries available. And I don't think that Boost would be ever able to provide something close to it. Because it just a set of great small libraries and not single bit framework.
and at the same time complaining about the compiled size of Boost.ASIO just fails the consistency test...
Just for the record: simple HTTP+SCGI+FCGI server that does trivial things has about 1MB code for logic only compiled with ASIO. I don't think this is can be good for ASIO. Qt that provides... Lets see full GUI toolkit and includes all functionality that boost has+ GUI, opengl, sql, webkit browser in 40MB is not bad at all. All is matter of how do you look on this.
Furthermore, even the libraries you mentioned, like Qt, do really only of the time interval...And since we are dealing with C++ here, your much bigger problem is the nonexisting ABI guarantees of the language itself...in the general case, if you upgrade or change your compiler bye bye ABI compatiblity...if you really want a stable ABI you'd have to go for a C one...
As I mentioned before - GCC keeps its ABI for 6 years, MSVC keeps ABI in average for 2-3 years. It is far better then 3 month that boost has.
As has already been suggested, why don't you just simply link statically with Boost?
Static linking has huge issues: - Updates require full recompilation - Each small application that need regex would add 0.7M to its size. How much bloat do you have! I don't see any reason today use anything but shared libraries. We in 2010.
You must realize that what you are asking from Boost is simply not feasible and seems reasonable only to you or a subset of Boost users...
I'm not asking Boost to become Qt-like library I'm asking to create an ABI stable subset of most important Boost libraries for example all in tr1 may become ABI stable easily.
For example, if I understood you correctly, you would even like to have shared_ptr<> pimpled...
Take a look on following two files: sp_counted_base.h http://cppcms.svn.sourceforge.net/viewvc/cppcms/framework/branches/refactoring/booster/booster/smart_ptr/sp_counted_base.h?revision=1172&view=markup sp_counted_base.cpp http://cppcms.svn.sourceforge.net/viewvc/cppcms/framework/branches/refactoring/booster/lib/smart_ptr/src/sp_counted_base.cpp?revision=1185&view=markup It is the only class I had to change in boost-shared ptr to make it ABI stable. How did I this? 1) I moved implementation to library 2) I added pthread_mutex_t to the class even if I do not use it. 3) I removed debug hooks. Now what happens: 1. I can change the implementation of the counter - use atomic-ops assembly or OS specific operations without warring about compatibility because all operations are done in single library. 2. I can add atomic operations on platforms where they were not availible before without changing layout of class - just by stopping using pthread_mutex_t. What does it requires? 1. Linking against some core library. 2. Little thinking about what can happen. Nothing more! Just to be aware of possible ABI issues.
In other words...if you dislike the Boost 'philosophy' so much, why do you even use it..?
Boost has great philosophy - evolution rather then design. But when some library becomes de-facto standard library in C++ world and everybody uses it because it is evolved in something great I expect it become stable, maintainable library with responsibility to non-bleeding-edge users that boost is targeted for them today.
...most of its functionality can be found in other libraries (like ACE instead of ASIO...)...
1. Boost has better design (I love ASIO API, but I hate compilation times and bloat) 2. Boost has standardized API - so when you talk about shared_ptr it is THE shared_ptr 3. Boost has many great libraries (like iostream). Rephrasing Stoustrup's saying: There are two kinds of libraries - that everybody hates - that nobody uses And sometimes I really hate boost, but it is impossible to leave without it in modern C++ world. Artyom
Artyom wrote:
Heavy template code produces significantly lager code. So hiding implementation actually reduces code size and reduces bloat.
Ridiculous. Misuse of templates can do so, but I doubt you'll find (m)any examples in Boost. Templates expose more code to the compiler permitting otherwise impossible optimizations. Expression templates, for example, eliminate tremendous runtime code and overhead utterly impossible to do otherwise.
Sometimes I just prefer to move some members to pimpl just in order to prevent inclusion of 10 more useless headers.
Preference.
Good example is Linux kernel. As policy it does not accept macros and inline functions in code. (Similar to templates in C++ or in-header implementations) because it reduces code size and thus significantly reduces cache misses.
Generalizations are useless. Profiling is the only way to know which particular design and implementation decisions are appropriate for a given use case. Inlining can be overused easily, of course, but not doing so can be just as bad.
If Linux was written for Modern C++ compiler with "Modern" C++ design in Boost style, it would probably would not be able to run on computers with several MB of memory.
Unsubstantiated conjecture.
As has already been suggested, why don't you just simply link statically with Boost?
Static linking has huge issues:
- Updates require full recompilation
That's a huge issue?
- Each small application that need regex would add 0.7M to its size. How much bloat do you have!
How much does the alternative regex library you use add to an application?
I don't see any reason today use anything but shared libraries. We in 2010.
There are many good reasons to use static libraries, but that's irrelevant to this discussion.
You must realize that what you are asking from Boost is simply not feasible and seems reasonable only to you or a subset of Boost users...
I'm not asking Boost to become Qt-like library I'm asking to create an ABI stable subset of most important Boost libraries for example all in tr1 may become ABI stable easily.
That isn't Boost's purpose, so it is highly unlikely to occur.
For example, if I understood you correctly, you would even like to have shared_ptr<> pimpled...
Take a look on following two files:
sp_counted_base.h
http://cppcms.svn.sourceforge.net/viewvc/cppcms/framework/bran ches/refactoring/booster/booster/smart_ptr/sp_counted_base.h?r evision=1172&view=markup
sp_counted_base.cpp http://cppcms.svn.sourceforge.net/viewvc/cppcms/framework/bran ches/refactoring/booster/lib/smart_ptr/src/sp_counted_base.cpp ?revision=1185&view=markup
It is the only class I had to change in boost-shared ptr to make it ABI stable.
How did I this?
1) I moved implementation to library 2) I added pthread_mutex_t to the class even if I do not use it. 3) I removed debug hooks.
So, you increased the size, decreased the performance, and removed functionality. The result isn't shared_ptr. It is possible that shared_ptr could have been designed with conditional compilation to choose a header-only or shared library implementation, but that would have been harder to maintain and would have prevented interoperation between applications and libraries trying to mix them.
Now what happens:
1. I can change the implementation of the counter - use atomic-ops assembly or OS specific operations without warring about compatibility because all operations are done in single library. 2. I can add atomic operations on platforms where they were not availible before without changing layout of class - just by stopping using pthread_mutex_t.
Of course that's all possible, but you aren't using boost::shared_ptr.
In other words...if you dislike the Boost 'philosophy' so much, why do you even use it..?
Boost has great philosophy - evolution rather then design. But when some library becomes de-facto standard library in C++ world and everybody uses it because it is evolved in something great I expect it become stable, maintainable library with responsibility to non-bleeding-edge users that boost is targeted for them today.
So, you like that Boost races along the bleeding edge of C++ development, producing advanced, even state-of-the-art designs that are considered for standardization, but you'd also like it to be stable. I'm confused. I get that it would be nice to have maintenance releases that include backported bug fixes so that one could continue to use older releases for a longer period rather than chasing the latest release to get bug fixes. That would ensure longer lived ABI compatibility. That much I can see being considered. The rest is just antithetical to Boost.
2. Boost has standardized API - so when you talk about shared_ptr it is THE shared_ptr
Except when you rewrite it and pretend the result is still boost::shared_ptr. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
Ridiculous. Misuse of templates can do so, but I doubt you'll find (m)any examples in Boost. Templates expose more code to the compiler permitting otherwise impossible optimizations. Expression templates, for example, eliminate tremendous runtime code and overhead utterly impossible to do otherwise.
Take a look on this message: http://lists.boost.org/Archives/boost/2010/03/163468.php
- Updates require full recompilation
That's a huge issue?
Yes when you have 3rd part library without sources with bug in old version of static library it linked with which was solved long time ago but you can't change it because it uses static library. What would happen of libstdc++ was statically linked or even libc? Bad things. And boost becomes today as important as libstdc++.
- Each small application that need regex would add 0.7M to its size. How much bloat do you have!
How much does the alternative regex library you use add to an application?
I mean that I prefer to use shared libraries - they reduce memory use, bloat because they are actually shared. So adding shared library virtually does not add any overhead.
How did I this?
1) I moved implementation to library 2) I added pthread_mutex_t to the class even if I do
not use it.
3) I removed debug hooks.
So, you increased the size, decreased the performance, and removed functionality. The result isn't shared_ptr.
How did I decreased performance? By calling function instead of in-lining it? Just for the record atomic ops cost hundreds of cycles so function call is negligible. Not talking about a fact that every unit uses same code cached in memory. How did I removed functionality (this is really interesting?)
It is possible that shared_ptr could have been designed with conditional compilation to choose a header-only or shared library implementation, but that would have been harder to maintain and would have prevented interoperation between applications and libraries trying to mix them.
You can always have boost::abi::shared_ptr
Of course that's all possible, but you aren't using boost::shared_ptr.
[SNIP]
2. Boost has standardized API - so when you talk about shared_ptr it is THE shared_ptr
Except when you rewrite it and pretend the result is still boost::shared_ptr.
boost::shared_ptr, std::tr1::shared_ptr, std::shared_ptr and my booster::shared_ptr is same pointer as it has same interface and same semantics - that what is important. Exactly as STLPort's auto_ptr, libstdc++'s auto_ptr, apache stdcxx's auto_ptr is same auto_ptr with same semantics. I don't see any difference. Artyom
Artyom wrote:
Ridiculous. Misuse of templates can do so, but I doubt you'll find (m)any examples in Boost. Templates expose more code to the compiler permitting otherwise impossible optimizations. Expression templates, for example, eliminate tremendous runtime code and overhead utterly impossible to do otherwise.
Take a look on this message:
Asio addressed such issues in 1.43. Has that resolved the problem you mentioned? Furthermore, did you test performance of the two designs? That is, what was the performance cost of using pimpls versus not? It is often reasonable to trade space for performance. Saying that a header only library can lead to code bloat is one thing. Blaming it on templates generally is another. You can argue whether header only is appropriate for a particular use case, but it has its merits, too.
- Updates require full recompilation
That's a huge issue?
Yes when you have 3rd part library without sources with bug in old version of static library it linked with which was solved long time ago but you can't change it because it uses static library.
That's not a problem with the approach I described previously.
What would happen of libstdc++ was statically linked or even libc?
Bad things.
And boost becomes today as important as libstdc++.
Boost isn't designed to be a single, system-wide source of functionality for a wide array of libraries and applications. To do so would make it something else entirely.
- Each small application that need regex would add 0.7M to its size. How much bloat do you have!
How much does the alternative regex library you use add to an application?
I mean that I prefer to use shared libraries - they reduce memory use, bloat because they are actually shared.
So adding shared library virtually does not add any overhead.
Boost.Regex builds a shared library.
How did I this?
1) I moved implementation to library 2) I added pthread_mutex_t to the class even if I do not use it. 3) I removed debug hooks.
So, you increased the size, decreased the performance, and removed functionality. The result isn't shared_ptr.
How did I decreased performance? By calling function instead of in-lining it? Just for the record atomic ops cost hundreds of cycles so function call is negligible. Not talking about a fact that every unit uses same code cached in memory.
I didn't look at the code you changed. Perhaps the specific change you made was as appropriate as you describe. I merely commented on the fact that you moved code out of line that I assumed Peter had inlined appropriately for performance reasons.
How did I removed functionality (this is really interesting?)
If you remove functionality, you change the design. Whether you think the removed functionality is "really interesting" is not the point.
It is possible that shared_ptr could have been designed with conditional compilation to choose a header-only or shared library implementation, but that would have been harder to maintain and would have prevented interoperation between applications and libraries trying to mix them.
You can always have boost::abi::shared_ptr
Someone has to create and maintain it.
Of course that's all possible, but you aren't using boost::shared_ptr.
[SNIP]
2. Boost has standardized API - so when you talk about shared_ptr it is THE shared_ptr
Except when you rewrite it and pretend the result is still boost::shared_ptr.
boost::shared_ptr, std::tr1::shared_ptr, std::shared_ptr and my booster::shared_ptr is same pointer as it has same interface and same semantics - that what is important.
Exactly as STLPort's auto_ptr, libstdc++'s auto_ptr, apache stdcxx's auto_ptr is same auto_ptr with same semantics.
I don't see any difference.
Just as was mentioned before, you cannot have different act-alike shared_ptrs share the same memory because they don't manage the reference count the same way. Therefore, you must copy the data, rather than share it, when moving from one to another. auto_ptr's don't share data. Using the same type to copy means transferring ownership. Transferring from one auto_ptr implementation to another requires manual intervention. They don't work well together. There is a significant difference. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
Someone has to create and maintain it.
I think Artyom started the conversation by identifying a perceived issue and volunteering to be involved in creating and maintaining a solution.
Erik
To be honest, I'm quite thinking to volunteer. But few points: 1. I would start it little bit later (I need to get my CppCMS beta release available and finish my Boost.Locale submission). 2. I would not be able to use Boost-Build. It is too weak to configure various system options correctly. I think that boost-stable should be configured in compilation time, and not with huge boost/config/* So probably CMake would be only option. 3. I would probably need to get some help from original library developers. 4. I would probably have to drop support of old or buggy compilers. i.e. use gcc>=3.4 MSVC>=8 not lower. 5. I have to be sure that my time would not be wasted for nothing and Boost community would reject the whole idea. Best, Artyom
On 5/18/2010 10:27 AM, Artyom wrote:
2. I would not be able to use Boost-Build. It is too weak to configure various system options correctly.
I think that boost-stable should be configured in compilation time, and not with huge boost/config/*
So probably CMake would be only option.
Pre-configuration is, IMO, going backwards. If I have to rely on running some tool to get to the code I want to use it's *much* less likely I'll use that code. I've passed over many libraries, and I tend to use a large variety of them, because they required pre-configuration. Real portability means not requiring pre-configuration! -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org (msn) - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail
Pre-configuration is, IMO, going backwards. If I have to rely on running some tool to get to the code I want to use it's *much* less likely I'll use that code.
I understand these reasons for header only libraries but for any compiled libraries you have to build them in any case. Also it is unlikely that ABI stable libraries would be header-only probably only very few one.
Real portability means not requiring pre-configuration!
Unfortunately when your write cross platform code for multiple compilers and libraries it is impossible to do this with any kind of configuration. (See the size of boost/config) However, compile time configuration is generally much more reliable then any other "defines-only" based. I have had issues when some important parts of Boost where broken on arm platform because of lack of proper configuration. Simple compile time checks like "check_cxx_source_compiles" would save lots of headache in system configuration. Not talking about the fact the single config.hpp file would save lots of compilation time instead of browsing over tens files in boost/config/* directory. Note: Boost has lots of header only libraries, Boost::abi would probably have almost any library compiled, so it is reasonable require compile time configuration. Artyom
Artyom wrote:
Pre-configuration is, IMO, going backwards. If I have to rely on running some tool to get to the code I want to use it's *much* less likely I'll use that code.
Unfortunately when your write cross platform code for multiple compilers and libraries it is impossible to do this with any kind of configuration.
(See the size of boost/config)
And yet, Boost.Config does just that.
However, compile time configuration is generally much more reliable then any other "defines-only" based.
I have had issues when some important parts of Boost where broken on arm platform because of lack of proper configuration.
Relying on platform-specific manifest constants can prevent using certain features due to choosing conservative options by default, and code can fail spectacularly by choosing liberal options by default. Knowing the correct value by introspection is certainly beneficial.
Simple compile time checks like "check_cxx_source_compiles" would save lots of headache in system configuration. Not talking about the fact the single config.hpp file would save lots of compilation time instead of browsing over tens files in boost/config/* directory.
I doubt there would be much compile time savings by moving from tens of files to one. However, there is value in having an autoconfig-style tool that uses tests to determine platform capabilities and produce a report of needed configuration options. Such a tool could even include the boost/config headers to verify correct settings. It would be terrific for verifying Boost.Config and for determining settings needed for new platforms versus the guessing and trial-and-error approach now used.
Note: Boost has lots of header only libraries, Boost::abi would probably have almost any library compiled, so it is reasonable require compile time configuration.
I assume you mean configuration information from a separate configure step when you use the phrase, "compile time configuration." (Boost.Config provides compile-time configuration, so the phrase is confusing.) I don't understand why compiled libraries should make a separate configuration step necessary, better, or even more acceptable. The same information is supplied either way. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
However, there is value in having an autoconfig-style tool that uses tests to determine platform capabilities and produce a report of needed configuration options. Such a tool could even include the boost/config headers to verify correct settings. It would be terrific for verifying Boost.Config and for determining settings needed for new platforms versus the guessing and trial-and-error approach now used.
There is a configure script for Boost.Config already, and yes it can check the existing options and report what needs to change as well. See http://www.boost.org/doc/libs/1_43_0/libs/config/doc/html/index.html#boost_c... and http://www.boost.org/doc/libs/1_43_0/libs/config/doc/html/index.html#boost_c... John.
At Wed, 19 May 2010 07:19:13 -0400, Stewart, Robert wrote:
...there is value in having an autoconfig-style tool that uses tests to determine platform capabilities and produce a report of needed configuration options.
This is one argument for trying to switch to CMake, which has that functionality. -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com
On 5/19/2010 11:19 AM, David Abrahams wrote:
At Wed, 19 May 2010 07:19:13 -0400, Stewart, Robert wrote:
...there is value in having an autoconfig-style tool that uses tests to determine platform capabilities and produce a report of needed configuration options.
This is one argument for trying to switch to CMake, which has that functionality.
And so does Boost Build now.. or at least close to it. We are in the process of implementing the Boost Test config script with the new BB functionality. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org (msn) - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail
On 5/19/2010 2:50 PM, Gennadiy Rozental wrote:
Rene Rivera<grafikrobot<at> gmail.com> writes:
And so does Boost Build now.. or at least close to it. We are in the process of implementing the Boost Test config script with the new BB functionality.
Is this related to Boost.Test or something else?
Oops.. I meant Boost Config :-\ -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org (msn) - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail
...there is value in having an autoconfig-style tool that uses tests to determine platform capabilities and produce a report of needed configuration options.
As we do already... we could easily hook up the existing configure script to Boost.Build, or for that matter we could use the new configuration logic in Boost.Build to do everything within that. I just haven't seen a huge demand for it, John.
What would happen of libstdc++ was statically linked or even libc?
Bad things.
And boost becomes today as important as libstdc++.
Boost isn't designed to be a single, system-wide source of functionality for a wide array of libraries and applications. To do so would make it something else entirely.
With more and more functionality being added to boost, the likelyhood for people to try using it for exactly that purpose increases. I do not think this is a bad thing, nor do I think it goes completely contrary to boost's evolution. But due to boost's design, it cannot be used for that purpose in the same way as it is possible for libstdc++ or libc. For both libstdc++ and libc, ABI changes were generally avoided as far as possible. Boost doesn't do so, nor do I think it would do boost anything good if it did. But using appropriate tools not to avoid ABI multiplicity, but to deal with it, it might well become a system-wide functionality source. Such tools could be: * user-controlled name mangling to avoid collisions * ABI introspection (like a static int boost::shared_ptr<T>::abi_version) * explicit indirection facilities to shield important interfaces against ABI changes * smarter build autoconfiguration tools Best regards, Isidor
Asio addressed such issues in 1.43. Has that resolved the problem you mentioned?
No I didn't yet.
Furthermore, did you test performance of the two designs? That is, what was the performance cost of using pimpls versus not? It is often reasonable to trade space for performance.
The difference does not exits. Why? Because the major cost in such libraries are system calls that are heavy. Inlining does not really help unless there are very few instructions in the code.
Artyom wrote:
Furthermore, did you test performance of the two designs? That is, what was the performance cost of using pimpls versus not? It is often reasonable to trade space for performance.
The difference does not exits. Why? Because the major cost in such libraries are system calls that are heavy. Inlining does not really help unless there are very few instructions in the code.
I'd like to see proof of your assertion. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
On 18/05/10 13:30, Artyom wrote:
boost::shared_ptr, std::tr1::shared_ptr, std::shared_ptr and my booster::shared_ptr is same pointer as it has same interface and same semantics - that what is important.
It is plain wrong. Interface and specification of std::tr1::shared_ptr is significantly different from std::shared_ptr, so both state very different types in terms of definition and behaviour. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net
Stewart, Robert wrote:
I get that it would be nice to have maintenance releases that include backported bug fixes so that one could continue to use older releases for a longer period rather than chasing the latest release to get bug fixes. That would ensure longer lived ABI compatibility.
It's not easy to maintain ABI compatibility while making changes. libstdc++ has dedicated tests for that, and they routinely do earn their keep.
Peter Dimov wrote:
Stewart, Robert wrote:
I get that it would be nice to have maintenance releases that include backported bug fixes so that one could continue to use older releases for a longer period rather than chasing the latest release to get bug fixes. That would ensure longer lived ABI compatibility.
It's not easy to maintain ABI compatibility while making changes. libstdc++ has dedicated tests for that, and they routinely do earn their keep.
Of course. I meant, and should have written, "that would *help to* ensure longer lived ABI compatibility." Indeed, it could be policy that no ABI breaking changes can be made, even for the sake of bug fixes, in a maintenance release. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
I get that it would be nice to have maintenance releases that include backported bug fixes so that one could continue to use older releases for a longer period rather than chasing the latest release to get bug fixes. That would ensure longer lived ABI compatibility. That much I can see being considered. The rest is just antithetical to Boost.
Well, boost's design makes it particularly hard to fix all too much without breaking the ABI. So, libraries that want to be ABI-stable will likely want to put an indirection layer around code which uses boost. What I am wondering is if there would be some interest in the boost community to build such an indirection layer (or building blocks thereof). Best regards, Isidor
Stewart, Robert <Robert.Stewart <at> sig.com> writes:
Artyom wrote:
Heavy template code produces significantly lager code. So hiding implementation actually reduces code size and reduces bloat.
Ridiculous. Misuse of templates can do so, but I doubt you'll find (m)any examples in Boost. Templates expose more code to the compiler permitting otherwise impossible optimizations. Expression templates, for example, eliminate tremendous runtime code and overhead utterly impossible to do otherwise.
I do not think this is ridiculous. This is well known fact. Templates do lead to the larger code size and larger compilation time. You can hardly argue with that. The same time I agree that they lead to better performance in many cases. Not always. I had number of real life scenarios where excessive inlining lead to losing locality and eventually worse performance. You always need to balance these two forces. If you get 0.5% performance advantage, but required to add half an half an hour to the compilation, your choice is pretty much clear, right? An ability to define this boundary between what to put in headers and what to offline correctly is important designer quality. Gennadiy
Gennadiy Rozental wrote:
Stewart, Robert <Robert.Stewart <at> sig.com> writes:
Artyom wrote:
Heavy template code produces significantly lager code. So hiding implementation actually reduces code size and reduces bloat.
Ridiculous. Misuse of templates can do so, but I doubt you'll find (m)any examples in Boost. Templates expose more code to the compiler permitting otherwise impossible optimizations. Expression templates, for example, eliminate tremendous runtime code and overhead utterly impossible to do otherwise.
I do not think this is ridiculous. This is well known fact. Templates do lead to the larger code size and larger compilation time. You can hardly argue with that.
Template code must, of necessity be duplicated in any shared library or application using it with the same type, to be sure. However, discounting that form of size increase, there is nothing about templates that would cause bloat except failure to factor out non-dependent code. Non-dependent code, of course, must then be inlined to be header only, and *that* can lead to code bloat. The foregoing, however, does not translate to "template code produces significantly larger code."
The same time I agree that they lead to better performance in many cases. Not always. I had number of real life scenarios where excessive inlining lead to losing locality and eventually worse performance.
Of course. Premature optimization is often pessimization. It's hard to get right.
You always need to balance these two forces. If you get 0.5% performance advantage, but required to add half an half an hour to the compilation, your choice is pretty much clear, right?
An ability to define this boundary between what to put in headers and what to offline correctly is important designer quality.
Quite right. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
On 19 May 2010 21:49, Gennadiy Rozental <rogeeff@gmail.com> wrote:
I do not think this is ridiculous. This is well known fact. Templates do lead to the larger code size and larger compilation time. You can hardly argue with that.
*Instantiated* templates lead to code size. Non-instantiated ones are a parsing cost, but nothing else. Compared to a runtime collection where everything is provided through some string factory interface, in the usual case of using, say, 2 hash algorithms of the dozen provided, the template code can easily be smaller.
The same time I agree that they lead to better performance in many cases. Not always. I had number of real life scenarios where excessive inlining lead to losing locality and eventually worse performance.
Excessive inlining is an optimizer issue orthogonal to template usage.
At Wed, 19 May 2010 19:49:33 +0000 (UTC), Gennadiy Rozental wrote:
Templates do lead to the larger code size and larger compilation time. You can hardly argue with that.
I certainly can. It all depends how they are used. If you use other means of polymorphism (e.g. virtual functions) on a fine-grained level, the cost in code size of dispatching to trivial functions that could otherwise be inlined can easily overwhelm “template code bloat.” http://www.cpptalk.net/image-vp103267.html#103267 Not that I'm saying a networking library would be designed that way... -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com
On 19 May 2010 18:54, David Abrahams <dave@boostpro.com> wrote:
At Wed, 19 May 2010 19:49:33 +0000 (UTC), Gennadiy Rozental wrote:
Templates do lead to the larger code size and larger compilation time. You can hardly argue with that.
I certainly can. It all depends how they are used. If you use other means of polymorphism (e.g. virtual functions) on a fine-grained level, the cost in code size of dispatching to trivial functions that could otherwise be inlined can easily overwhelm “template code bloat.”
In that conversation you say that: "As far as I know, getting everything just right involves walking the line between static and dynamic polymorphism, compiling some generic things down to object code, and leaving others to be "instantiated" inline." I think nobody can object to that, and the discussion/problem is about where to draw the line. But I could have misunderstood =) What I find strange is that Boost librariy developers, in general, seems to agree that template bloat is of no or little concern, #include dependencies is of no or little concern, and compilation time for end-user is of absolutely no concern. A few developers bothers with this, and their work I appreciate very much. The one -big- concern is that every piece of code is visible so that it can be inlined in every compilation unit. For performance. Why so? Performance is a very complicated matter as everybody knows, and inlining is no holy grail. Performance is improved by analysis of a running program and taking proper action, but the more time I sit around waiting for compilation to finish after minor modification the less time I can spend on such activities. It would, I think, be beneficial to Boost if the standpoint on header only was less strict, and that ideas about hiding implementation details in .cpp files were not looked upon as craziness. =) Looking forward to a wrapped up Boost btw, if Booster ever comes to life! Cheers, Christian
What I find strange is that Boost librariy developers, in general, seems to agree that template bloat is of no or little concern, #include dependencies is of no or little concern, and compilation time for end-user is of absolutely no concern.
You just said what I was thinking about. There is some kind of Urban Myths in Boost community - Headers always better then sources. - Inline functions is best way to improve performance. - Concepts are **always** better then inheritance (even if they triple the size of the code). Sad.
Looking forward to a wrapped up Boost btw, if Booster ever comes to life!
Just a small point: I really hope that someday there would be boost-stable, boost-abi or just boost would become stable. I think it would be great if that stability would be official and supported part of original boost. I don't think that Booster is the way to go because I created it for my partial needs rather needs of Boost community. For example: - Booster.Thread is just wrapper of pthreads/pthreads-win32 with boost like API. - Booster.Regex is just a wrapper of PCRE. I had no time to start adopting Boost code for ABI stability I have interests of my project. That are first at this point. ABI stable boost should be part of Boost and not external boost-like library. In any case you are wellcome to try it and give any inputs: As I told it has, regex, threads, aio, function, smart-pointers and more. Booster is generally description of the reasons I need to create one. Artyom
On 20 May 2010, at 17:38, Artyom wrote:
What I find strange is that Boost librariy developers, in general, seems to agree that template bloat is of no or little concern, #include dependencies is of no or little concern, and compilation time for end-user is of absolutely no concern.
You just said what I was thinking about. There is some kind of Urban Myths in Boost community
- Headers always better then sources. This one is a matter of person taste but... - Inline functions is best way to improve performance. I've found this to be true in my own work. So have lots of other people. Prove us wrong. - Concepts are **always** better then inheritance (even if they triple the size of the code). What do you mean by better? They are more flexable, and often perform better. If you disagree, prove us wrong.
If you could produce code which had the same performance, was more flexable and compiled faster, I am certain many people would be happy to use it. Perhaps there is also space in C++ for more "boring" libraries, which just aim to implement a solid ABI-stable interface in a clean way. There is no reason such libraries could not exist. But they are not particularly interesting to work on, so it can be hard to motivate people to donate their time to them for no benefit. If there was demand, backed by substantial amounts of money, for a boost branch which was ABI-stable, I am sure someone would produce one. Chris
Hi,
- Inline functions is best way to improve performance. I've found this to be true in my own work. So have lots of other people. Prove us wrong.
Let me have a go at describing some bloat. It's not in my interest to prove anything. I'll just tell you cost we see in real-life applications. On Linux, GCC 4.5.x, x86_64, we have executables which load 598 DSO images mapped in 611 memory regions, corresponding to: - 263'431'108 100.0% bytes mapped from shared libraries - 261'638'634 99.3% bytes total allocated to sections - 1'792'474 0.7% bytes padding not in any section (= rounding to page) The break down by sections that are actually loaded into memory: - 114'086'965 43.3% code (.text) - 65'205'187 24.8% dynamic symbols and related tables - 26'825'486 10.2% unwind tables - 24'356'624 9.2% plt + relocations + related tables - 19'000'712 7.2% global data - 11'325'928 4.3% global common data (.bss) - 749'576 0.3% various shared library headers - 82'138 0.0% global constructors and destructors - 5'890 0.0% glibc memory management voodoo - 128 0.0% thread-specific data That's ~55% "real stuff", ~25% of symbol tables, ~10% unwind tables, ~10% relocations and PLT. The application virtual memory size is about a gigabyte, so this is a major fraction of the overall footprint. There are 544'533 symbols which represent 142'548'100 bytes. Of this there are 272'190 weak symbols, or 43'565'063 bytes. A significant fraction of those weak symbols represent template duplication across libraries, but that's not the only form of bloat we see. There are 2'599 symbols with at least 10 duplicates, total 5'832'419 bytes, and 118 vtables with at least 10 duplicates (about 300k). So over half of the symbols and about a third of the size are ill-advicedly generated inline functions, virtual function tables (19'043 vtables = 2'802'928 bytes) and type info objects and names (45'961 typeinfo objs + names = 3'142'851 bytes). This goes with accompanying symbol tables, PLTs, unwind tables, and so on. A significant fraction of the 60+ MB symbol tables is obviously for long mangled names. There is a very significant number of tiny (5-7 byte) functions which are actually just stub that call the real function via PLT. If function A calls B, we actually have A call out-of-line stub function which calls PLT entry which jumps to B. And this is mostly for inline functions that should never have been inline in the first place! Unsurprisingly we see a lot of CPU stalls linked to instruction prefetch issues. We used to see massive scale TLB failures as well (over 60% of L2 cache accesses were for code some time back), but with wider TLBs there's less of that. Intel's performance experts have indicated to us directly that the amount of code we have is a real challenge and reducing the amount of code, and better code locality, are likely required to improve performance. I am certain there are projects where "headers only" is nice. There are also projects where the entire package is written into the header files, on the assumption the compiler or build system will sort out where to put the code, generates massive additional costs. (Boost is a contributor to the above bloat, but not the top one.) Regards, Lassi
- Inline functions is best way to improve performance. I've found this to be true in my own work. So have lots of other people. Prove us wrong.
Let me have a go at describing some bloat. It's not in my interest to prove anything. I'll just tell you cost we see in real-life applications.
Thanks for providing some real numbers - much better than endless speculation! If there are any big offenders that are Boost libraries then lets hear about it: I'm sure we can take it ;-)
On Linux, GCC 4.5.x, x86_64, we have executables which load 598 DSO images mapped in 611 memory regions, corresponding to:
- 263'431'108 100.0% bytes mapped from shared libraries - 261'638'634 99.3% bytes total allocated to sections - 1'792'474 0.7% bytes padding not in any section (= rounding to page)
The break down by sections that are actually loaded into memory:
- 114'086'965 43.3% code (.text)
Ouch, that's one big application.
- 65'205'187 24.8% dynamic symbols and related tables - 26'825'486 10.2% unwind tables - 24'356'624 9.2% plt + relocations + related tables - 19'000'712 7.2% global data - 11'325'928 4.3% global common data (.bss) - 749'576 0.3% various shared library headers - 82'138 0.0% global constructors and destructors - 5'890 0.0% glibc memory management voodoo - 128 0.0% thread-specific data
That's ~55% "real stuff", ~25% of symbol tables, ~10% unwind tables, ~10% relocations and PLT. The application virtual memory size is about a gigabyte, so this is a major fraction of the overall footprint.
There are 544'533 symbols which represent 142'548'100 bytes. Of this there are 272'190 weak symbols, or 43'565'063 bytes.
A significant fraction of those weak symbols represent template duplication across libraries, but that's not the only form of bloat we see. There are 2'599 symbols with at least 10 duplicates, total 5'832'419 bytes, and 118 vtables with at least 10 duplicates (about 300k).
Are you able to use separate file template instantiation to reduce duplication?
So over half of the symbols and about a third of the size are ill-advicedly generated inline functions, virtual function tables (19'043 vtables = 2'802'928 bytes) and type info objects and names (45'961 typeinfo objs + names = 3'142'851 bytes). This goes with accompanying symbol tables, PLTs, unwind tables, and so on.
A significant fraction of the 60+ MB symbol tables is obviously for long mangled names.
That is one big problem with templates that I'll freely admit to. Compiler vendors are certainly aware of this, and some of them have reduced mangled name size over the years, but I guess it'll always be an issue :-( Regards, John.
Hi,
If there are any big offenders that are Boost libraries then lets hear about it: I'm sure we can take it ;-)
Let's see, let me start with these. Need out-of-line versions of repeat instantiations everywhere: - boost::noncopyable_::noncopyable -> need out-of-line typeinfo - boost::detail::sp_counted_base::destroy() - boost::detail::sp_counted_base -> need out-of-line dtor, vtable, typeinfo - boost::detail::shared_count -> need out-of-line dtor, vtable, typeinfo - boost::system::system_error -> need out-of-line dtor, vtable, typeinfo - boost::system::error_category -> need out-of-line dtor, vtable, typeinfo - boost::system::error_category::equivalent() - boost::system::error_category::default_error_condition() - boost::system::error_code::unspecified_bool_true() - boost::system::system_error::what() const - boost::exception_detail::clone_base -> need out-of-line dtor, vtable, typeinfo - boost::thread_exception -> need out-of-line dtor, vtable, typeinfo - boost::lock_error -> need out-of-line dtor, vtable, typeinfo - boost::thread_resource_error -> need out-of-line dtor, vtable, typeinfo - boost::bad_function_call::bad_function_call() Need to eliminate objects generated into every source file - boost::system::system_category - boost::system::posix_category - boost::system::native_ecat - boost::system::generic_category - boost::system::errno_ecat - boost::tuples::ignore - boost::lambda::detail::(anonymous namespace)::constant_null_type If your reaction is "surely those will be generated inline / elided by compiler", well, nope they didn't in our code.
Ouch, that's one big application.
Actually that was a small one :-) It only pulls in ~600 DSO. We have 3000.
A significant fraction of those weak symbols represent template duplication across libraries, but that's not the only form of bloat we see. There are 2'599 symbols with at least 10 duplicates, total 5'832'419 bytes, and 118 vtables with at least 10 duplicates (about 300k).
Are you able to use separate file template instantiation to reduce duplication?
This is duplication across shared libraries, not within a single library. I've tried to push us to build a handful of large libraries instead of 3000 mostly small ones, but given other constraints that will be a multi-year project. (Not your fault of course.)
So over half of the symbols and about a third of the size are ill-advicedly generated inline functions, virtual function tables (19'043 vtables = 2'802'928 bytes) and type info objects and names (45'961 typeinfo objs + names = 3'142'851 bytes). This goes with accompanying symbol tables, PLTs, unwind tables, and so on.
A significant fraction of the 60+ MB symbol tables is obviously for long mangled names.
That is one big problem with templates that I'll freely admit to. Compiler vendors are certainly aware of this, and some of them have reduced mangled name size over the years, but I guess it'll always be an issue :-(
That one way to look at it, yes, and I look forward to seeing the fruits of that. I personally just tend to encourage developers to understand what compilers in the real world do, and adapt code to that, and set conventions to match. It's of course a moving target. But you cannot just splat all the code into header and hope the compiler does something smart with that, it will just not produce good results in a big system. Same of course applies to any bad idea, like returning vector<int> by value, having a std::map inside for loop, or having code for std::map<..., std::map<..., std::vector<SomethingComplex>>> inside some object that has compiler-generated copy ctor, with everything declared inline. That's just a recipe to generate 1MB machine code and a memory allocation storm whenever someone decides to make a copy. In general I find people are over-optimistic about what the compiler will really do to your code. I'll be first to say there are projects where the above don't matter. But there are projects where it does. Regards, Lassi
Hi,
Let's see, let me start with these. Need out-of-line versions of repeat instantiations everywhere:
Oh BTW, these were the easy / trivial ones, i.e. non-templates. Re full symbol lists please see my other message. I could get them if you want, but it will take some time as I am about to go off on some travel. If you want to be amused, one of the most common template ones that bite us by generating tons of stuff all over the place is __PRETTY_FUNCTION__ inside the boost::shared_ptr's assertions :-) Alas we can't enable -DNDEBUG for all our software, there are useful assertions elsewhere. Regards, Lassi
Lassi Tuura wrote:
If you want to be amused, one of the most common template ones that bite us by generating tons of stuff all over the place is __PRETTY_FUNCTION__ inside the boost::shared_ptr's assertions :-)
Alas we can't enable -DNDEBUG for all our software, there are useful assertions elsewhere.
You can disable just BOOST_ASSERT with BOOST_DISABLE_ASSERTS though. But I suspect you don't want to do that either because you're using BOOST_ENABLE_ASSERT_HANDLER, or you wouldn't be getting the __PRETTY_FUNCTION__s. Patch current_function.hpp then. :-)
Hi,
Oh BTW, these were the easy / trivial ones, i.e. non-templates. Re full symbol lists please see my other message. I could get them if you want, but it will take some time as I am about to go off on some travel.
Some quick examples for templates: - boost::intrusive_ptr<T>::~intrusive_ptr(), at least 15 kB in numerous instances, I guess includes dtor cost for T in some cases. - boost::detail::sp_counted_impl_p<T>::~sp_counted_impl_p(), at least 10kB total in lots of 15-byte functions. - boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<T> >::~clone_impl() and "non-virtual thunk to", also at least 20kB via hundred+ small bits (max about 100 bytes each). - boost::shared_ptr<T>::operator=(boost::shared_ptr<T> const&) also at least 10kB. - typeinfo and vtable for boost::detail::sp_counted_impl_p<T> tens of kilobytes (50 or more). That's just a quick sampler. Any more thorough stats will have to be weeks from now. Lassi
Lassi Tuura wrote:
Some quick examples for templates:
- boost::intrusive_ptr<T>::~intrusive_ptr(), at least 15 kB in numerous instances, I guess includes dtor cost for T in some cases. - boost::detail::sp_counted_impl_p<T>::~sp_counted_impl_p(), at least 10kB total in lots of 15-byte functions. - boost::exception_detail::clone_impl<boost::exception_detail::e rror_info_injector<T> >::~clone_impl() and "non-virtual thunk to", also at least 20kB via hundred+ small bits (max about 100 bytes each). - boost::shared_ptr<T>::operator=(boost::shared_ptr<T> const&) also at least 10kB. - typeinfo and vtable for boost::detail::sp_counted_impl_p<T> tens of kilobytes (50 or more).
All of those are, presumably, for different T's, right? Can that be counted as duplication? Not if the destructor code is dependent on T, which a delete would be (and is most of those functions). _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
Hi,
All of those are, presumably, for different T's, right? Can that be counted as duplication? Not if the destructor code is dependent on T, which a delete would be (and is most of those functions).
No. There are different types, but the bulk of it is repetition across libraries. The more visible the type, the more copies. Something very common will get instantiated into 500 libraries, and even if it's 100 bytes each, or even just 10 bytes, it adds up quickly. For example for any specific type ABC, boost::detail::sp_counted_impl_p<ABC>::~sp_counted_impl_p() is 15 bytes only, but we have 76 of them, so it's actually 1140 bytes in executable memory (only 15-byte version gets used), plus function alignment padding, plus unwind tables, plus symbol names, plus relocation tables, plus PLT entries, and on and on. One way to look at it is templates in interface is not a good idea if you will build more than a handful of DSOs. If you have hundreds or thousands like we do, templated types tend to generate bloat at dramatic pace. Now why would functions declared inline get emitted 15-byte out-of-line functions, well, there's tons of practical issues behind that. But all those bigger and smaller "practical issue" streams add up to 140 MB for us. Regards, Lassi
Hi,
- boost::detail::shared_count -> need out-of-line dtor, vtable, typeinfo
shared_count needs a vtable? Are you sure?
Sorry, the comment was from slightly more general group of symbols that needed at least one of the above ("needs vtable" meant needs *out-of-line* vtable if it had one). So in this case it needs just out-of-line destructor. Thanks for pointing that out. We don't want you to go add vtables :-) I am just telling you what is duplicated. FWIW, boost::detail::shared_count::~shared_count() is our 50th largest symbol by combined size, 474 * 109 bytes = 51666 bytes in total. It's also 24th most common symbol and 4th most common weak symbol. There's actually another similarly named function (demangles to same name), which must be compiler-generated clone with different symbol name, at position 14987, 1484 bytes in total (106 instances times 14 bytes). boost::detail::shared_count::shared_count(boost::detail::shared_count const&) is 11661 largest symbol, 1888 bytes (118 * 16). Regards, Lassi
At Thu, 20 May 2010 19:23:55 +0200, Lassi Tuura wrote:
Hi,
- Inline functions is best way to improve performance.
I've found this to be true in my own work. So have lots of other people. Prove us wrong.
Let me have a go at describing some bloat. It's not in my interest to prove anything. I'll just tell you cost we see in real-life applications.
<snip> Lassi, I just want to thank you, deeply, for posting real data. Beats speculative claims every time. Cheers! -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com
Hi,
I just want to thank you, deeply, for posting real data. Beats speculative claims every time.
You are welcome, I've collected tons of data. I do have full symbol lists sorted by combined size and frequency for our apps. Although the code's open source, I'd have to check if I can send you the full lists, would probably have to ask scientific publication requirements that apply to our software to get clearance... Besides they are big :-) Maybe later if you want. Regards, Lassi
On May 20, 2010, at 1:23 PM, Lassi Tuura wrote:
- Inline functions is best way to improve performance. I've found this to be true in my own work. So have lots of other people. Prove us wrong.
Let me have a go at describing some bloat. It's not in my interest to prove anything. I'll just tell you cost we see in real-life applications.
On Linux, GCC 4.5.x, x86_64, we have executables which load 598 DSO images mapped in 611 memory regions, corresponding to:
[... snipped detailed breakdown ...] That's ~55% "real stuff", ~25% of symbol tables, ~10% unwind tables, ~10% relocations and PLT. [...]
There are 544'533 symbols which represent 142'548'100 bytes. Of this there are 272'190 weak symbols, or 43'565'063 bytes.
Any idea how much of the symbol space would go away with proper application of symbol visibility, such as defaulting to hidden and only exposing those that are needed? I realize that's hard to measure without actually having gone through the exercise of getting all that stuff done right, which may be quite hard in some cases (especially when dealing with third-party code). See the currently ongoing thread with subject "GCC symbol visibility across shared libraries". A while ago I did some work on reducing the size of our boost.python-based wrapper libraries. I wasn't able to make any progress using the symbol visibility controls, as there was too much underlying work that needed to be done. However, I was able to achieve a > 50% reduction in the size of these libraries by using linker scripts that encoded a few simple heuristics.
Hi,
There are 544'533 symbols which represent 142'548'100 bytes. Of this there are 272'190 weak symbols, or 43'565'063 bytes.
Any idea how much of the symbol space would go away with proper application of symbol visibility, such as defaulting to hidden and only exposing those that are needed? I realize that's hard to measure without actually having gone through the exercise of getting all that stuff done right, which may be quite hard in some cases (especially when dealing with third-party code). See the currently ongoing thread with subject "GCC symbol visibility across shared libraries".
We are looking into it, but I don't have the numbers, sorry. We know enough that it's interesting; there were some snags and it was disabled again. We are currently only looking setting visibility of *inlines* to hidden, looking at more extensive controls is a much longer-term project. Unfortunately the inline visibility controls in GCC only affect class members, it doesn't control free template functions (from memory, but something like that). It helped quite a bit, but not nearly as much as I had hoped for. But it does help our start-up times, less time spent for relocations and so on. Regards, Lassi
What I find strange is that Boost librariy developers, in general, seems to agree that template bloat is of no or little concern, #include dependencies is of no or little concern, and compilation time for end-user is of absolutely no concern.
Of course it's a concern - some of us do try and minimise template instantiation - there was even a tool for profiling template instantiations posted around here somewhere. Some of us also provide C API's for things like Boost.Regex and Boost.Math so that folks that are only concerned with "trivial" use cases can gain faster compilation times. Some of us also provide simple (fast to compile) forward declaration headers that omit any template implementations - the forward header can be used in most places to speed compilation, while instantiating used templates only in one place - a good way to keep track of which templates are actually getting instantiated as well. Having said that, precompiled headers almost completely negate the additional cost of #including header only libraries anyway.
You just said what I was thinking about. There is some kind of Urban Myths in Boost community
- Headers always better then sources.
Because it's something we're often asked for. There are folks who have told me that they won't use Boost.Regex precisely because it has separate source - even though that reduces code bloat and yes compilation times too for common use cases (the most heavily used template are instantiated in the library and *not* in your object files. There should be no more code bloat than for a non-template library *unless* you misuse it.
- Inline functions is best way to improve performance. - Concepts are **always** better then inheritance (even if they triple the size of the code).
Sad.
No not sad at all - we should all use the best tool for the job in hand. If that's inheritance so be it, if that's template so be it. To pick two extremes - a GUI library is a perfect case for inheritance, where as Math lib is IMO a perfect case for templates. In both cases if the implemenation and design are poorly done then the result can be frustration. But that's no excuse such sweeping statements.
I really hope that someday there would be boost-stable, boost-abi or just boost would become stable. I think it would be great if that stability would be official and supported part of original boost.
I don't think that Booster is the way to go because I created it for my partial needs rather needs of Boost community.
For example:
- Booster.Thread is just wrapper of pthreads/pthreads-win32 with boost like API.
Oh please no!!! Why should I be forced to use a GPL'ed external library rather than a "first class citisen" that talks to the platform API's directly.
- Booster.Regex is just a wrapper of PCRE.
Oh heavens. So lets see... no wide character support, no support for segmented containers (think really large texts stored in discontinous memory - I've had people using this support to search multi-gigabyte texts that could never be stored in memory or searched via a C interface). BTW do you realise that Boost.Regex has had an ABI stable POSIX-standard compatible C interface for the last um.... 12 years.... and yes that's since before Boost existed. Apologies if this has turned into a rant, but there are so many misconceptions and myths here that I really didn't know where to begin. I'll end with this thought: "templates generate no more code bloat than non-templates unless they are used gratuitously". For example instantiating vector<int> and vector<long> in the same program on a platform where int and long are the same size should probably be considered a mistake. But that's no reason to ditch std::vector, just to be sensible in ones usage. Regards, John.
John Maddock wrote:
For example instantiating vector<int> and vector<long> in the same program on a platform where int and long are the same size should probably be considered a mistake.
I'd call that a QoI issue, though being aware of such problems may prove necessary as Lassi has shown. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
First of Boost I didn't proposed Booster to be replacement of boost as I don't really care about full Windows support. So don't take a look on Booster as what Boost should be.
I don't think that Booster is the way to go because I
my partial needs rather needs of Boost community.
For example:
- Booster.Thread is just wrapper of
created it for pthreads/pthreads-win32 with boost
like API.
Oh please no!!!
Why should I be forced to use a GPL'ed external library rather than a "first class citisen" that talks to the platform API's directly.
1. pthreads-win32 is LESSER GPL library (not GPL) and many-many project use it 2. Boost.Thread is just C++ implementation of Pthreads standard. Every function that exists in Boost.Threads exists in pthreads and other way. Boost thread is designed AFTER pthreads concepts and API.
- Booster.Regex is just a wrapper of PCRE.
Oh heavens. So lets see... no wide character support,
IMHO wide characters should be vanished from C++ it is the worst way to represent Unicode (and I know thing or two about Unicode). See: - http://stackoverflow.com/questions/1049947/should-utf-16-be-considered-harmf... - http://cppcms.sourceforge.net/boost_locale/html/tutorial.html#myths And PCRE supports Unicode quite well with UTF-8 support. And if you really want Unicode - only ICU can help you.
no support for segmented containers (think really large texts stored in discontinous memory - I've had people using this support to search multi-gigabyte texts that could never be stored in memory or searched via a C interface).
Ahhhhh... Are you running regex over multi-gigabyte texts? Then something wrong with your design.
BTW do you realise that Boost.Regex has had an ABI stable POSIX-standard compatible C interface for the last um.... 12 years.... and yes that's since before Boost existed.
No I do not. To be honest I'm glad to hear it (really...) I'll take a look of I can extract Boost.Regex as is for Booster. And it would be easy to put it into boost-abi
I'll end with this thought: "templates generate no more code bloat than non-templates unless they are used gratuitously". For example instantiating vector<int> and vector<long> in the same program on a platform where int and long are the same size should probably be considered a mistake. But that's no reason to ditch std::vector, just to be sensible in ones usage.
Nobody tells you make vector non-template or non-inlined as vector is just type-safe array with iterators just wrappers of pointers. But I do not think that networking library should be inlined (for example) It is the point of design. Best, Artyom
- Booster.Regex is just a wrapper of PCRE.
Oh heavens. So lets see... no wide character support,
IMHO wide characters should be vanished from C++ it is the worst way to represent Unicode (and I know thing or two about Unicode).
See:
- http://stackoverflow.com/questions/1049947/should-utf-16-be-considered-harmf... - http://cppcms.sourceforge.net/boost_locale/html/tutorial.html#myths
And PCRE supports Unicode quite well with UTF-8 support. And if you really want Unicode - only ICU can help you.
Which is supported by Boost.Regex as well. For Win 32 work though, you simply cannot escape wchar_t whatever you may think of it - this is especially true of the embedded windows platforms which are effectively wchar_t only.
no support for segmented containers (think really large texts stored in discontinous memory - I've had people using this support to search multi-gigabyte texts that could never be stored in memory or searched via a C interface).
Ahhhhh... Are you running regex over multi-gigabyte texts?
Then something wrong with your design.
Not me, but I know folks who are. And no there's nothing wrong with their design - if you a multi-gigabyte text on disk (from some other source outside your control) you have to handle it *somehow*. Sure you could use partial-matches, but they're only an approximation to what you really want to do. John.
- Booster.Regex is just a wrapper of PCRE.
Oh heavens. So lets see... no wide character support,
IMHO wide characters should be vanished from C++ it is the worst way to represent Unicode (and I know thing or two about Unicode). For Win 32 work though, you simply cannot escape wchar_t whatever you may think of it - this is especially true of the embedded windows platforms which are effectively wchar_t only.
[SNIP] Some notes... Booster is library build especially for CppCMS purposes and not generic library for Boost replacements. I do not suggest anybody use Booster as Boost replacement. I just use to show important point: 1. CppCMS what to provide *stable* ABI 2. CppCMS needs Boost Goodies 3. CppCMS can't use Boost, because Boost is ABI *unstable* 4. Because of these issues I was forced to create Booster. that is Boost like library that covers small subset of Boost tools with limited functionality, but it fits CppCMS project needs. And this is bad... Bad for me and bad for Boost and bad for any other library that thinks of making their ABI stable. Thus I demonstrate as the title of this entire discussion: --------------------------------------- Boost is useless for library developers --------------------------------------- Who may be "crazy enough" to think that stable ABI is good thing. Best Regards, Artyom
On 05/20/2010 12:38 PM, Artyom wrote:
What I find strange is that Boost librariy developers, in general, seems to agree that template bloat is of no or little concern, #include dependencies is of no or little concern, and compilation time for end-user is of absolutely no concern.
You just said what I was thinking about. There is some kind of Urban Myths in Boost community
- Headers always better then sources. - Inline functions is best way to improve performance. - Concepts are **always** better then inheritance (even if they triple the size of the code).
Each time has its dogmas and zealots. What used to be the OO hype now seems to have become the above, at least in the corner of the world that this community is occupying. One of the best things about C++ is that it supports many differen programming paradigms. Choosing the right one(s) for a particular problem domain is definitely not a trivial task. (It may even involve hybrid programming, i.e. involve multiple languages deployed together.) Stefan -- ...ich hab' noch einen Koffer in Berlin...
On 20 May 2010 18:38, Artyom <artyomtnk@yahoo.com> wrote:
You just said what I was thinking about. There is some kind of Urban Myths in Boost community
- Concepts are **always** better then inheritance
Sad.
I don't consider this one an urban myth. - If you need different types, then inheritance doesn't work feasibly, so concepts are the only choice, and are thus better by default - If inheritance can be feasibly done, then a single virtualizing wrapper can be written for the concept. Since the overhead of this is just in the virtual tables and such, it's the same as would be added in making it natively virtual (and all the wrapping instantiation of templates can be done in source files, making the templating completely invisible to the users of the virtual interface, and meaning that there's no opportunity for multiple-instantiation "bloat"). So since it allows additional options at negligible -- if any -- cost, it's better. Unless you have a counterexample?
At Thu, 20 May 2010 10:16:03 -0500, Christian Holmquist wrote:
On 19 May 2010 18:54, David Abrahams <dave@boostpro.com> wrote:
At Wed, 19 May 2010 19:49:33 +0000 (UTC), Gennadiy Rozental wrote:
Templates do lead to the larger code size and larger compilation time. You can hardly argue with that.
I certainly can. It all depends how they are used. If you use other means of polymorphism (e.g. virtual functions) on a fine-grained level, the cost in code size of dispatching to trivial functions that could otherwise be inlined can easily overwhelm “template code bloat.”
In that conversation you say that: "As far as I know, getting everything just right involves walking the line between static and dynamic polymorphism, compiling some generic things down to object code, and leaving others to be "instantiated" inline."
I think nobody can object to that, and the discussion/problem is about where to draw the line. But I could have misunderstood =)
Perhaps; perhaps not. I just think “template code bloat” is overhyped as a problem and generally poorly understood, so when people make unqualified statements about it like Gennadiy's above, I like to wave a dowsing rod at it.
What I find strange is that Boost librariy developers, in general, seems to agree that template bloat is of no or little concern,
I don't. You can see that from the design of Boost.Python, which assiduously tries to get everything that can be static into the library binary.
#include dependencies is of no or little concern, and compilation time for end-user is of absolutely no concern.
I don't know any Boost developers who believe those things.
A few developers bothers with this, and their work I appreciate very much.
Some have been better at dealing with it than others.
The one -big- concern is that every piece of code is visible so that it can be inlined in every compilation unit. For performance. Why so?
Performance is a distinguishing feature of C++ among languages in which high-level programming is possible, so we naturally have lots of users who care a lot about performance. Once you trade away performance for something else, you can't get it back. On the other hand, if you use too little dynamic polymorphism, you can end up with huge executables that blow the instruction cache. It's a balancing act.
Performance is a very complicated matter as everybody knows, and inlining is no holy grail. Performance is improved by analysis of a running program and taking proper action, but the more time I sit around waiting for compilation to finish after minor modification the less time I can spend on such activities.
I know :(. Try Clang; it compiles boost now and is wicked fast. :)
It would, I think, be beneficial to Boost if the standpoint on header only was less strict,
There is no strict standpoint on it. Different developers have different positions on the issue.
and that ideas about hiding implementation details in .cpp files were not looked upon as craziness. =)
Looking forward to a wrapped up Boost btw, if Booster ever comes to life!
Booster? I musta missed that thread. -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com
Looking forward to a wrapped up Boost btw, if Booster
ever comes to life!
Booster? I musta missed that thread.
It is a small "boost-like" ABI stable library I had to create for my project CppCMS, as ABI stability is one of its important goals. https://cppcms.svn.sourceforge.net/svnroot/cppcms/framework/branches/refacto... Artyom
[I still lurk on the list, although I almost never post these days... sorry] Artyom, all: I feel compelled to chime in, because while maintaining the Bloomberg STL and core application infrastructure between 2006--2009, I've also seen my share of crazy stuff. With hundreds of thousands of objects linked into more than a thousand libraries, compilation times were a real concern. In addition, but this is peculiar to Bloomberg infrastructure (and for historical reasons), some executables had to link in great amounts of stuff. Sometimes the game was to try to keep the size of the debug symbols below the limit (as large as 2G on some compilers), other times it was the size of the code segment (which could reach up to 1.5G). At times it did seem futile, but when the company's business depends on it, and a seemingly innocuous change added tens of MB just because the debug symbols grow, and you cannot roll out the next release, these were real problems. I must say, boost could be used in the leaves of the libraries hierarchy, but was forbidden in any "core" library. I myself was not part of that decision from management, but the reasons as I understood them were that we had components with similar functionality and slightly better performance (because tuned and optimized for our use cases), but mostly I think because of the header-only approach of many boost libraries. Although we did use templates, we were careful to limit the amount of inlining and did template hoisting every chance we got. We had reimplemented a large part of boost (granted, with our own support for allocators, Ion knows something of it since he and Pablo Halpern had long discussions about the scoped allocator model). We attempted to factor as much code as possible into the static libs, and keep the headers for truly small inline functions. For instance, we had a partial specialization of vector<T*> that was a thin wrapper around vector<void*> (template hoisting), and basic_string<char> in terms of a non-template class. I think that templates can be a problem, but if that is a concern, there are techniques to mitigate that. Namely, do not allow templates to propagate. Templates are for foundational libraries (boost::function, boost::variant, std::vector, std::map, boost::shared_ptr). In application domains, use concrete types. If I am writing a map from a set of symbols to a set of values (let's say an attribute class), that I am going to use all over my code base, I would certainly consider writing an interface (or set of interfaces) whose implementation would use, but not export, the types std::map<symbol, boost::tuple<attribute_1, attribute_2, ... > >. Thus clients (that is, my code base) would simply use MySymbolMap. All the templates would reside in MySymbolMap.o and that's it. There are also advantages to that in terms of readability. For instance, vector<MyClass> becomes MyClassVector. In the first phase, it can be a typedef, but having the type defined separately and used as MyClassVector throughout the code base gives the option to later define it as a thin wrapper, and move functions out-of-line so that translation units that include it no longer have to reference vector<MyClass>. Likewise, since tuples and variant can be template hogs, it was frowned upon to use typedef to instantiations of the variant template, but encouraged instead to define a concrete type for each tuple or variant instantiation. So unless I have a very good reason to keep templates, when I am writing application-specific code, I try to factor my types in a single translation unit, within which I use template tools for the implementation. This is a well-known technique, but unfortunately, if you don't apply it systematically, template instantiation creeps in and it's a pain to refactor along it. Of course, this presupposes that there is some kind of hierarchical approach to code design (and thus some kind of high-level architect - we had John Lakos). Speaking of that, and just as a case in point, I think at some point, the longest symbol in the code base was a whopping 10K long (mangled, iirc) on the IBM compiler (which has a poor mangling scheme, unlike the SunPro compiler which does some sort of subexpression factoring for which this symbol was only a few KBs). If memory serves, it was some kind of
"std::map< std::map<std::string, std::pair<float, MyClass> >::iterator, std::pair<std::string, std::pair<double, MyClass> > >::iterator std::map< std::map<std::string, std::pair<float, MyClass> >::iterator, std::pair<std::string, std::pair<double, MyClass> > >::insert(std::map< std::map<std::string, std::pair<float, MyClass> >::iterator, std::pair<std::string, std::pair<double, MyClass> > >::iterator, std::pair<std::string, std::pair<double, MyClass> >::iterator, std::pair<std::string, std::pair<double, MyClass> > > const&);"
or something like this with, but a few more levels of nestedness in the pairs and maps. You might think it's crazy, but it's actually not that hard to do with code like: struct MyClass struct FloatMap { typedef std::pair<float, MyClass> ValueClass; typedef std::map<std::string, ValueClass> SymbolMap; typedef SymbolMap::value_type SymbolValue; typedef SymbolMap::iterator iterator; }; struct DoubleMap { typedef std::pair<Value, MyClass> ValueClass; typedef std::map<std::string, ValueClass> SymbolMap; typedef SymbolMap::value_type SymbolValue; typedef SymbolMap::iterator iterator; }; std::map<FloatMap::iterator, DoubleMap::SymbolValue> myMap; myMap.insert(...); Modeling relationships between financial instruments can involve several levels of maps, and that's absolutely legit. The point is that, instead of defining his own types, the developer kept defining them using typedefs and std:: facilities like map, pair, and iterator, and there were several levels of that in different files. Each file by itself looked nice and concise. But at the end, he ended using it up within a function template (or inline, can't remember which), which was instantiated within dozens of translation units. Thus the occurrence of that symbol in several (too many) object files. -- Hervé Brönnimann hervebronnimann@mac.com
Hi all, Sometimes I have used a variant of PImpl idiom that hides all the private functions and private static data of a class (not a template) on a XImpl class. Even if this doesn't hides the data, it reduces considerably the compilation time and the dependencies on the header files. One of the advantages of the approach is that it not incurs in allocation/deallocation of the XImpl class as this class has no other data than static one. The other is that your header file doesn't change when you refactor your implementation adding, removing or modifying private functions or private static data (as far as you don't need to change the member data, of course). Note that this schema don't forbids the compiler for inlining on the implementation file. You can see an usage example here http://viboes.blogspot.com/2010/04/pimpl-variant.html. Of course this is not applicable to templates, which are the big part of most of the Boost libraries, and it doesn't solves the ABI problem neither. Any comments are welcome :) Best, _____________________ Vicente Juan Botet Escribá http://viboes.blogspot.com/
David Abrahams <dave <at> boostpro.com> writes:
At Thu, 20 May 2010 10:16:03 -0500, Christian Holmquist wrote:
On 19 May 2010 18:54, David Abrahams <dave <at> boostpro.com> wrote:
At Wed, 19 May 2010 19:49:33 +0000 (UTC), Gennadiy Rozental wrote:
Templates do lead to the larger code size and larger compilation time. You can hardly argue with that.
I certainly can. It all depends how they are used. If you use other means of polymorphism (e.g. virtual functions) on a fine-grained level, the cost in code size of dispatching to trivial functions that could otherwise be inlined can easily overwhelm “template code bloat.”
I was not trying to proof a theorem, but general trend (and my experience as well) is that template based solution in comparison with *semantically* identical solution based on runtime polymorphism usually takes longer to compile and produce larger code. Fun fact: in company I work for, the fear of code bloat and longer compilation/link time (admittedly based on some bad experience) lead to almost every instance of vector and shared_ptr to be explicitly instantiated. Hoverer much I dislike it I can't tell 200 developers "it's ugly - let's add 10 min to your compilation/link time".
What I find strange is that Boost librariy developers, in general, seems to agree that template bloat is of no or little concern,
I don't. You can see that from the design of Boost.Python, which assiduously tries to get everything that can be static into the library binary.
I myself is very heavy Boost.Python user. And in this case in spite of the fact that my code is probably longest to compile, I believe that for the most part Boost.Python got this balance right. Not everyone in a team agree with me though. Gennadiy
We also pay attention at that. In our case, any compialtiontime > 10s for some trivial to simpel code make our users frown. We spent a lot of time redesigning our lib so those times fall down. Then we sued Steven Watanabe profiler to get even better. I don't think "nobody cares about compile-time" is right. -- ___________________________________________ Joel Falcou - Assistant Professor PARALL Team - LRI - Universite Paris Sud XI Tel : (+33)1 69 15 66 35
I don't see any reason today use anything but shared libraries. We in 2010.
There are many good reasons to use static libraries, but that's irrelevant to this discussion.
I think it is relevant. So there ought to be a different set of libraries for each version of gcc? For each combination of BOOST_NO_IOSTREAM, BOOST_NO_TYPEID and so on and so on? For the misguided sake of an ABI? I've used boost for quite a while but never been interested in using the generic shared libraries. - Nigel
Artyom wrote:
Take a look on following two files:
sp_counted_base.h
You don't need the mutex. Look at the recent sp_counted_base_spin.hpp.
sp_counted_base.cpp http://cppcms.svn.sourceforge.net/viewvc/cppcms/framework/branches/refactoring/booster/lib/smart_ptr/src/sp_counted_base.cpp?revision=1185&view=markup
No need to use atomic_set in the constructor. If your constructor races with another member, atomics won't save you.
It is the only class I had to change in boost-shared ptr to make it ABI stable.
How did I this?
1) I moved implementation to library 2) I added pthread_mutex_t to the class even if I do not use it. 3) I removed debug hooks.
Now what happens:
1. I can change the implementation of the counter - use atomic-ops assembly or OS specific operations without warring about compatibility because all operations are done in single library. 2. I can add atomic operations on platforms where they were not availible before without changing layout of class - just by stopping using pthread_mutex_t.
What does it requires?
1. Linking against some core library. 2. Little thinking about what can happen.
With all due respect, may I submit that the reason it was so easy to make shared_ptr ABI-stable was that it was designed with this in mind? shared_ptr was frequently criticised for having too much overhead and being too slow, and it was an intentional design decision to allow add_ref/release to be inlined. I knew that on most relevant architectures this didn't provide any significant performance benefit (even making them virtual is affordable - and this is another interesting tradeoff, allowing ABI-incompatible shared_ptr instances to coexist), but C++ programmers are notoriously picky. Besides, Boost was entirely header-only at the time shared_ptr was first designed and implemented, and had no build system. In any event, the ABI-stable shared_ptr is spelled std::shared_ptr.
You don't need the mutex. Look at the recent sp_counted_base_spin.hpp.
I just preferred to fail to general mutex when atomic ops are missing. In any case it looks like this happens only on Linux-non-x86 platforms and on Linux mutex is already userspace operation (thanks to futexes) I did this mostly for simplistically hopefully that no-implementations would use mutex finaly.
No need to use atomic_set in the constructor. If your constructor races with another member, atomics won't save you.
I know. It was just for simplicity. Not much hard.
With all due respect, may I submit that the reason it was so easy to make shared_ptr ABI-stable was that it was designed with this in mind?
Yes indeed :-) it felt this. Just boost-version is not provide this ABI stability.
shared_ptr was frequently criticised for having too much overhead and being too slow, and it was an intentional design decision to allow add_ref/release to be inlined. I knew that on most relevant architectures this didn't provide any significant performance benefit
This is fully reasonable because in reality calls are much cheaper then any atomic operation.
(even making them virtual is affordable - and this is another interesting tradeoff, allowing ABI-incompatible shared_ptr instances to coexist),
That is really interesting. It would indeed solve the ABI issues even without creating separate library. Thus each library would use the shared_ptr ABI created by the original object creator.
but C++ programmers are notoriously picky.
Or... Have too many prejudices like virtual functions are heavy or inlining solves every possible performance problem that is not true.
Besides, Boost was entirely header-only at the time shared_ptr was first designed and implemented, and had no build system.
I think that for boost as-is today it is more then fine to have header only shared_ptr
In any event, the ABI-stable shared_ptr is spelled std::shared_ptr.
Amen!!! If most of compilers I want to support had C++0x library I would not have to use 80% of boost I use today. Artyom. P.S.: Thanks for looking to my code.
Artyom <artyomtnk <at> yahoo.com> writes:
...most of its functionality can be found in other libraries (like ACE instead of ASIO...)...
1. Boost has better design (I love ASIO API, but I hate compilation times and bloat)
ABI issues aside and not to step on anyone foots, but I personally would never use ASIO in my production code regardless how cool API is. There is no reason for the networking library to be implemented in headers (I myself developed 3 different async IO libraries for different companies, so I do have experience in this domain). I pretty sure you can have the same API with 99% of implementation hidden offline. Gennadiy
On Wed, May 19, 2010 at 12:36 PM, Gennadiy Rozental <rogeeff@gmail.com> wrote:
Artyom <artyomtnk <at> yahoo.com> writes:
...most of its functionality can be found in other libraries (like ACE instead of ASIO...)...
1. Boost has better design (I love ASIO API, but I hate compilation times and bloat)
ABI issues aside and not to step on anyone foots, but I personally would never use ASIO in my production code regardless how cool API is. There is no reason for the networking library to be implemented in headers (I myself developed 3 different async IO libraries for different companies, so I do have experience in this domain). I pretty sure you can have the same API with 99% of implementation hidden offline.
To be fair, I take the different view. There is absolutely no reason to make a network library *not* header-only. Putting everything in headers lets me, the library writer to never ever have to worry about preserving ABI because, well, there's no ABI to preserve across releases. To each his own I guess. ;) -- Dean Michael Berris deanberris.com
Dean Michael Berris <mikhailberis <at> gmail.com> writes:
Putting everything in headers lets me, the library writer to never ever have to worry about preserving ABI because, well, there's no ABI to preserve across releases.
How having simple API with hidden implementation lead to ABI problems? Gennadiy
On Wed, May 19, 2010 at 1:02 PM, Gennadiy Rozental <rogeeff@gmail.com> wrote:
Dean Michael Berris <mikhailberis <at> gmail.com> writes:
Putting everything in headers lets me, the library writer to never ever have to worry about preserving ABI because, well, there's no ABI to preserve across releases.
How having simple API with hidden implementation lead to ABI problems?
Well, because your return types might not be trivial and these returned types may have to inherit from some other types that need to be known and need to be preserved across version releases. Of course that is a corner case that might be easily avoided if you only ever returned int's or primitive data types. And then you have to run into the cost of doing a pimpl, and/or the cost of doing virtual function calls when they're really just unnecessary. I'm not even talking about performance here, just the pure necessity of it. If things can and should be done statically, there is no good reason to introduce an additional pointer or a vtable -- even if that reason is to preserve ABI compatibility, because in my head that's not a valid reason. And no, I don't think having pimpls all around will lead to ABI problems -- but it is something that I would rather avoid with just having a header-only library. Because as a library developer, I *don't* want to deal with ABI issues and just be able to release header-only libraries that just work without having to worry about preserving compatibility with earlier releases. But that might just be me. ;) -- Dean Michael Berris deanberris.com
participants (27)
-
Alexander Churanov -
Artyom -
Christian Holmquist -
Christopher Jefferson -
David Abrahams -
Dean Michael Berris -
Domagoj Saric -
Gennadiy Rozental -
Gottlob Frege -
Hervé Brönnimann -
Isidor Zeuner -
joel falcou -
John Maddock -
Kim Barrett -
Lassi Tuura -
Mateusz Loskot -
Nelson, Erik - 2 -
Nigel Stewart -
Peter Dimov -
Rene Rivera -
Robert Ramey -
Roland Bock -
Scott McMurray -
Stefan Seefeld -
Stewart, Robert -
vicente.botet -
Vladimir Prus