Update on status of C++ 11 only Boost fork/modular distro
Dear all, You may remember me saying these past six months that I was working on the tools necessary to fork Boost into a C++ 11 only distro which also lets you use Boost libraries individually standalone without the rest of Boost. I have got a first alpha version of my clang AST based local bindings generator working, and you can see it at https://github.com/ned14/local-bind-cpp-library. This tool lets you generate a set of local namespace bindings for some implementation of some library such that your library implementation code need not care what the underlying implementation of dependent library facilities is. This way your Boost library could be using either Boost or the C++ 11 STL and you don't need to care (most of the time). Now I have the first alpha version working and generating a reasonable set of C++ 11 STL bindings (https://github.com/ned14/local-bind-cpp-library/tree/master/include/s tl11), I'll start with porting over my own Boost libraries to it, so Spinlock, Expected, non-allocating future-promise and AFIO will be the first, and with those AFIO should become capable of being completely standalone from Boost whilst still also able to compile against Boost. After that I'll try porting the entire of Boost.Thread over such that Boost.Thread no longer needs Boost. I think that would be hugely useful for people wanting to use potential C++ 17 threading functionality without needing to draw in all of Boost as a dependency, and that might greatly widen the empirical experience new threading facilities might get before the 17 standard. Once I've succeeded with that, if the present situation with Boost has not improved by then, I'll press ahead with a new Boost distro website which score ranks Boost modules according to the scoring manifest I outlined on this list some months ago, and lets you download standalone Boost distros of your choice. Obviously only libraries ported to be standalone capable would be available, so I would hope that if I can demonstrate an easy port with Boost.Thread, others may follow with their libraries. All this work occurs in my free time of a few hours per week, so progress will be slow but steady. Comments on the local namespace binding tool are welcome. Comments on the proposed C++11/14 good practice idiom of binding dependent library APIs into the local namespace are also welcome, because as far as I can tell no one here has discussed a formal set of conventions to make best use of inline namespaces for new libraries. I'll put it another way - local namespace binds are a poor man's version of Herb's proposed C++ ABI facility. As much as they aren't quite as good, they are probably good enough (relative to Herb's most recent proposal). Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
2014-10-07 8:47 GMT-03:00 Niall Douglas <s_sourceforge@nedprod.com>:
Once I've succeeded with that, if the present situation with Boost has not improved by then, I'll press ahead with a new Boost distro website which score ranks Boost modules according to the scoring manifest I outlined on this list some months ago, and lets you download standalone Boost distros of your choice. Obviously only libraries ported to be standalone capable would be available, so I would hope that if I can demonstrate an easy port with Boost.Thread, others may follow with their libraries.
If the independent libraries will be "Boost-free", then what will their build system be? -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker
On 7 Oct 2014 at 11:46, Vinícius dos Santos Oliveira wrote:
Once I've succeeded with that, if the present situation with Boost has not improved by then, I'll press ahead with a new Boost distro website which score ranks Boost modules according to the scoring manifest I outlined on this list some months ago, and lets you download standalone Boost distros of your choice. Obviously only libraries ported to be standalone capable would be available, so I would hope that if I can demonstrate an easy port with Boost.Thread, others may follow with their libraries.
If the independent libraries will be "Boost-free", then what will their build system be?
Said libraries need to be header only, so no build system needed. Makes things much simpler. Some may observe that surely boost_system must be needed as it is a dependency of most header only Boost libraries, so you always end up having to find a copy from somewhere even with pure header only usage of Boost. Thanks to the namespace binder that goes away as it uses the STL system_error instead. My only remaining showstopper is Filesystem actually, if your library uses Filesystem you currently have little choice but to link the Boost Filesystem library except on MSVC which bundles a Filesystem implementation. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
2014-10-07 14:11 GMT-03:00 Niall Douglas <s_sourceforge@nedprod.com>:
Said libraries need to be header only, so no build system needed. Makes things much simpler.
Agreed. But what about unit tests? And how do we will install these libraries on the systems? Will the solution be embedding local versions of the libraries in the app? -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker
On 7 Oct 2014 at 15:44, Vinícius dos Santos Oliveira wrote:
Said libraries need to be header only, so no build system needed. Makes things much simpler.
Agreed. But what about unit tests?
Nothing in this fork effort stops you still using the exact same library within Boost, it just replaces Boost with the STL when standalone. Hence you'd still run the unit tests there. If you want unit testing capable of testing the standalone build, I am working on some macro shims based on CATCH which emulate Boost.Test. They won't be entirely equivalent though as CATCH isn't threadsafe (basically we won't call catch unless the test fails, so non-failing tests aren't logged). Library maintainers will have to port their unit test framework over, plus add cmake/Makefile as appropriate where necessary for standalone unit testing. Or hack their Jamfile, and have a standalone build tested from within Boost. Up to the maintainer.
And how do we will install these libraries on the systems?
git submodule add to your project.
Will the solution be embedding local versions of the libraries in the app?
You could, but adding it as a git submodule will probably be easier. My "custom build" script I expect to be a thin veneer of github's ability to zip git repos for you. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
Thanks for the answers Niall. I'll reserve some time to analyse your effort further. -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker
From a distribution point of view, I see some big problems too. I guess the idea, if I understand correctly, is to distribute your binding generator along with the select-few boost header-only libraries, and then, the user would have to run the binding generator for his particular platform to generate the implementation-detail binding headers of those boost library
Personally, I find this proposal very scary. I'm just gonna give my opinion on this, make what you want of it. One little thing, I'm not sure that STL11 is a particularly good name for this, since most of the new C++11 libraries would not be classified as "STL" libraries (except for unordered containers and <array>). But that's just a trivial issue of terminology. You state in your readme for your generator: ".. or write some brittle bindings which manually replicate STL items .." In my experience, what can make those kinds of manual bindings brittle are all those subtle incompatibilities (often undocumented) between analogous components of the alternative implementations. It's not hard to take a list of standard components and turn that into a list of using statements / typedefs / aliases within a new namespace, because such lists are not that long in general (20 to 50 things per header, at most) and it's not hard to do this well and to maintain them. I find that there's nothing particularly brittle about that "human process". Replacing that process with a reg-expr generated automatic process can only make it more brittle, with very few benefits, and might require significant tinkering until it works reliably, and I don't care how smart that generator is (using clang's AST lib or not), I cannot see this becoming more reliable than a hand-written and actively maintained equivalent. And after all that, it does not address the main weak point of such bindings, which is, as I said, dealing with the subtle incompatibilities and platform-specific quirks. ports. Or, if you want to spare the user from having to do this, then you must distribute a specific version of all the headers that will work for their particular platform (OS, compiler and std-lib implementation) and the particular versions of those platform components. Either way, this is a distribution nightmare. As for "forking" boost based on this, I don't understand why this is necessary, or maybe I don't quite understand what it is meant to really involve. Most boost libraries already contain a significant amount of conditional code and workarounds to deal with platform-specific issues or to use beneficial features that only some platforms support (including C++11 features). Why would it be necessary to have a completely separate version or distribution of those libraries that are targeted at platforms with the (mythical) feature-complete C++11 library implementation? That is, if you really mean to "fork" the boost libraries in that way, there is going to be enormous costs associated to doing that, mostly in terms of maintenance and man-power. I think that a much better alternative strategy is to do what boost has done with Boost.TR1. Why not create a library wrapper like Boost.TR1 but for C++11/14? Isn't that what users want, i.e., to be able to use standard C++11/14 library components and not have to worry about running their code on platforms that don't support it or only partially? I mean, people who use Boost.Thread today, instead of <thread>, use it because it works everywhere. But if they could include something like <boost/cpp11/thread.hpp>, and use "std::thread" and friends, then the only remaining platform-specific work would be to install and link against libboost_thread for platforms that do not provide the standard <thread> library. An additional way of doing this is to simply require that the maintainers of the libraries that have been standardized to update their library such that it binds to the standard C++11 library implementation (as thinly as possible) whenever available and sufficiently compatible (I think that most of the boost libraries are today, they seem to have recovered from all the mayhem following the C++11 standard and all its slight differences w.r.t. Boost). In other words, if you include <boost/thread.hpp> on a feature-complete C++11 platform, it should do little more than include <thread> and bind its components to the boost namespace. In both cases, the responsibility of maintaining the C++11/14 compatibility in those boost library rests on the shoulder of the individual maintainers. No forks required. And the local bindings (generated or not, from boost to std or vice versa) can be maintained more easily. Incompatibilities and other platform-specific can be dealt with more practically by manually maintaining the bindings. And users simply need to be advised about what features a platform should have to allow a specific boost library to be used standalone and / or without having to install and link to a pre-compiled library. Maybe I completely misunderstood your goals, and all I said are things you had already considered. I hope so. This is just the way I see things. Hearing the words "fork" in the same sentence as "boost", gives me chills... (unless it means "fork" only in the sense of a temporary developement branch that is meant to be merged back quickly). Mikael.
On 7 Oct 2014 at 16:55, Mikael Persson wrote:
Personally, I find this proposal very scary. I'm just gonna give my opinion on this, make what you want of it.
One little thing, I'm not sure that STL11 is a particularly good name for this, since most of the new C++11 libraries would not be classified as "STL" libraries (except for unordered containers and <array>). But that's just a trivial issue of terminology.
Interestingly, one of the most obvious things you see in the bindings output is the word 'template' due to how template heavy the C++ 11 additions are. But I mainly chose it for being short and descriptive.
You state in your readme for your generator: ".. or write some brittle bindings which manually replicate STL items .." In my experience, what can make those kinds of manual bindings brittle are all those subtle incompatibilities (often undocumented) between analogous components of the alternative implementations.
In the use case situation of providing some/any C++ 11 STL implementation, I would see Boost as just one of Dinkumware, libstdc++ and libc++. If your code already doesn't mind which STL it runs on, it is highly unlikely to get suddenly brittle with Boost, and moreover if it does, there is probably a bug in there. Of course, existing Boost libraries (with a few exceptions like ASIO) have only ever been tested against one C++ 11 STL implementation, the one coming with Boost. That is probably a bad thing. I would expect any surprises to emerge from binding to another C++ 11 STL to be a good thing. Even with AFIO, I found it broke in useful ways when I ported to libc++. That found sloppiness in my lack of use of noexcept which merely caused silent poor performance with libstdc++.
From a distribution point of view, I see some big problems too. I guess the idea, if I understand correctly, is to distribute your binding generator along with the select-few boost header-only libraries, and then, the user would have to run the binding generator for his particular platform to generate the implementation-detail binding headers of those boost library ports. Or, if you want to spare the user from having to do this, then you must distribute a specific version of all the headers that will work for their particular platform (OS, compiler and std-lib implementation) and the particular versions of those platform components. Either way, this is a distribution nightmare.
The idea is that the bindings generator makes a starting set of bindings. The library maintainer then adjusts those to fit their library and bundles them with their library. The bindings are then fixed, and are part of the library implementation. Remember that using Boost also goes through bindings, we basically allow the end user to choose which STL 11 implementation for a Boost library to use. I am proposing that we "virtualise" the STL implementation used by Boost libraries so it is end user selectable.
As for "forking" boost based on this, I don't understand why this is necessary, or maybe I don't quite understand what it is meant to really involve. Most boost libraries already contain a significant amount of conditional code and workarounds to deal with platform-specific issues or to use beneficial features that only some platforms support (including C++11 features). Why would it be necessary to have a completely separate version or distribution of those libraries that are targeted at platforms with the (mythical) feature-complete C++11 library implementation? That is, if you really mean to "fork" the boost libraries in that way, there is going to be enormous costs associated to doing that, mostly in terms of maintenance and man-power.
There is plenty of discussion in the archives of this mailing list to explain all that, and I would prefer to not repeat all that yet again. I will say this: this is part of one low cost path to achieving modular Boost. If you consider actual useful modularity desirable, this is one step along that path as somehow one needs to create separation between a Boost library and its dependencies, and it's not like we've been making progress on this for some years now other than endlessly debating header analysed dependencies, which while helpful doesn't actually achieve real modularisation. The local namespace binding technique lets the library maintainer explicitly mark up which dependencies are substitutable (e.g. STL ones), which are mandatory, and which are optional. We have yet to think of a formal set of guidelines for marking up mandatory dependencies and optional dependencies. Baby steps.
I think that a much better alternative strategy is to do what boost has done with Boost.TR1. Why not create a library wrapper like Boost.TR1 but for C++11/14? Isn't that what users want, i.e., to be able to use standard C++11/14 library components and not have to worry about running their code on platforms that don't support it or only partially? I mean, people who use Boost.Thread today, instead of <thread>, use it because it works everywhere. But if they could include something like <boost/cpp11/thread.hpp>, and use "std::thread" and friends, then the only remaining platform-specific work would be to install and link against libboost_thread for platforms that do not provide the standard <thread> library.
Thing is, TR1 was a set of new libraries, so it was straightforward to make a Boost.TR1 and it was contained. C++11/14 involves an enormous number of changes to *existing* Boost libraries, so we have been enhancing Boost libraries with C++ 11/14 features with varying rates of progress. Speaking only as someone on the Boost.Thread team as I can't speak much for other Boost libraries ... Boost.Thread already provides a significant superset of the C++ 11 STL, and the intent is that by next year it shall provide a significant portion of a potential C++ 17 threading STL too. However, we have a problem: what if a user library wants to use Boost.Thread's potential C++ 17 facilities with std:: not boost:: threading facilities - a perfectly reasonable request, especially if a user library only and exclusively uses std::thread et al and has zero wish to buy into the boost ecosystem apart from the 17 functionality? My opinion on this is that the potential C++ 17 features we add to Thread also need to be capable of existing in std::experimental, so as an example of a current work item of mine, the proposed expected<T, E> based non-allocating future-promise could appear in either or both of: 1. namespace boost { namespace expected_future { inline namespace v1 { ... 2. namespace std { namespace experimental { inline namespace expected_future { inline namespace v1 { ... However, if you think it through, expected_future::future<T, E> needs to understand both std::exception_ptr, std::chrono etc as well as boost::exception_ptr, boost::chrono etc. If a user configures a std::experimental target they may still wish to feed it boost STL items in addition to std STL items. They also may wish both expected_future implementations to coexist in the same code base, and to interoperate seamlessly. These are some of the issues we face moving forwards. C++ is complex, not helped by our collective historical failure to deal properly with ODR violation resolution or solving object componentisation as my C++ Now 2014 paper describes.
An additional way of doing this is to simply require that the maintainers of the libraries that have been standardized to update their library such that it binds to the standard C++11 library implementation (as thinly as possible) whenever available and sufficiently compatible (I think that most of the boost libraries are today, they seem to have recovered from all the mayhem following the C++11 standard and all its slight differences w.r.t. Boost). In other words, if you include <boost/thread.hpp> on a feature-complete C++11 platform, it should do little more than include <thread> and bind its components to the boost namespace.
Requiring maintainers to do anything, including do any maintenance, has proven to be hard. We already have many unmaintained libraries, some with decade old unfixed issues, so enforcing raised maintenance requirements just makes more unmaintained libraries. As a personal opinion, I would personally welcome that as an opportunity to deprecate and eliminate a large number of deteriorating Boost libraries, but mine is not the consensus opinion on this, and is definitely not the opinion of the steering committee.
In both cases, the responsibility of maintaining the C++11/14 compatibility in those boost library rests on the shoulder of the individual maintainers. No forks required.
At no stage did I ever propose the forked code stops being compatible with legacy monolithic Boost. Forked code simply becomes dual use. Why use the word fork then? It's still a fork in approach, methodology, intent, practice and most especially in Boost philosophy. It makes possible my alternative method of ranking library quality via automated scripts. That is definitely a fork in the definition of what qualifies a library to gain the Boost stamp of approval.
And the local bindings (generated or not, from boost to std or vice versa) can be maintained more easily. Incompatibilities and other platform-specific can be dealt with more practically by manually maintaining the bindings. And users simply need to be advised about what features a platform should have to allow a specific boost library to be used standalone and / or without having to install and link to a pre-compiled library.
Maybe I completely misunderstood your goals, and all I said are things you had already considered. I hope so.
I think what I am proposing actually is mostly what you are looking for.
This is just the way I see things. Hearing the words "fork" in the same sentence as "boost", gives me chills... (unless it means "fork" only in the sense of a temporary developement branch that is meant to be merged back quickly).
The other aspect of this being a fork is that new Boost libraries joining my modular distro simply wouldn't bother joining the monolithic Boost distro. That is already happening anyway to an extent - as my C++ Now presentation showed, exciting new C++ libraries are increasingly not trying to enter Boost any more and the Boost participation rate has been declining steadily for two years now. I am hoping that through modularation and my alternative automated ranking of quality, the archaic buy ins demanded by Boost to enter Boost go away and thus shall boost (!) the greenfield library participation rate. I in particular want the boost namespace to go away where possible, it screams monolithicism and cathedral instead of bazaar. And besides with inline namespaces and full fat namespace binding in 11/14, separate namespaces for everything is now very straightforward to do - let the user bind dependencies into their local namespace instead of Boost monolithically deciding for you. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
participants (3)
-
Mikael Persson -
Niall Douglas -
Vinícius dos Santos Oliveira