Hybrid compilation model? was: [system] Why is this not header-only?

There has been a thread running with the subject "[system] Why is this not header-only?" that discusses header-only versus compiled-library compilation models for Boost libraries. Some users require the compiled-library approach. I'll characterize these folks as requiring the shorter compile times and better software engineering practice that compiled libraries provide. In some cases these folks have had to rework Boost header-only libraries to create compiled libraries. Some users prefer the header-only approach. They need to be able to quickly set up at new project without messing around with compiled libraries. They want to be able to evaluate Boost libraries without doing any preparatory work altogether. Jeff Garland and Andrey Semashev argued that Boost should adopt a hybrid compilation model that allows applicable Boost libraries to be used in either header-only or compiled-library modes. Note that Boost.Test already provides this, although with degraded functionality in header-only form. There was some discussion as to how to do this; my eyes glazed over when discussions drifted into problems of dlls, singletons, etc. I'll like to challenge Boosters to think about this problem a bit more, and I'd love to see someone who understands the challenges take, say, Boost.System and demonstrate how it could be packaged for either header-only or compiled-library use. The important goal would be to abstract what to done into a general set of guidelines (and any configuration support needed). In other words, we don't so much need a solution for Boost.System as for any Boost library that would benefit from a hybrid computation model. Comments? --Beman

Beman Dawes wrote:
I'll like to challenge Boosters to think about this problem a bit more, and I'd love to see someone who understands the challenges take, say, Boost.System and demonstrate how it could be packaged for either header-only or compiled-library use. The important goal would be to abstract what to done into a general set of guidelines (and any configuration support needed). In other words, we don't so much need a solution for Boost.System as for any Boost library that would benefit from a hybrid computation model.
I'd do it, but I just dont' have time right now. Here's the basic details. BTW, this is detailed in Chapter 10 of Efficient C++ (Bulka/Mayhew) circa 2000 -- and I'm guessing they didn't invent it -- so, this isn't exactly a new idea. In their case the rational was performance tuning. Here's the basic idea: 1) in the header, conditionally include the implementation file (eg: .ipp) 2) in the implementation file, conditionally inline methods 3) in the .cpp file include .ipp and turn off the conditional inlining For system it appears that it's essentially error_code.hpp/cpp we need to convert. So here's what needs to be done. In error_code.hpp we add something like this at the near the end of the file: //make system all inline #ifdef BOOST_USE_OS_NATIVE_HEADERS_INLINE #include <error_code.ipp> #endif Now, in the IPP file you'll need a macro to optionally define inline: //code to define inline #ifdef BOOST_USE_OS_NATIVE_HEADERS_INLINE #define BOOST_INLINE inline #endif and the methods need to be declared like this: BOOST_INLINE int errno_ed( const error_code & ec ) To be safe at the end of the .ipp you'll probably want this: #ifdef BOOST_USE_OS_NATIVE_HEADERS_INLINE #undef BOOST_INLINE #endif Note that in system.cpp code looks like there's some namespace scope const arrays and things that probably need to be turned into enums or in general cleaned up for header only use. So the result is that when someone does #def BOOST_USE_OS_NATIVE_HEADERS_INLINE 1 #include <boost/system.hpp> the effect is all inlined code -- they don't need to link the library. Now the error_code.cpp effectively just includes the .ipp with the macro undefined. So it builds the functions into the library because BOOST_INLINE resolves to an empty string. So, for those that just use the usual: #include <boost/system.hpp> they will need to link the library. That's the essence of the technique...I'm sure someone with an hour or 2 to goof around with it could implement it on system. Jeff

On Fri, 06 Apr 2007 10:57:06 -0400, "Beman Dawes" <bdawes@acm.org> said:
There has been a thread running with the subject "[system] Why is this not header-only?" that discusses header-only versus compiled-library compilation models for Boost libraries.
Yes, I have also been skimming this thread with interest, but due to lack of time haven't been able to jump in. Thank goodness for long weekends :) [...]
I'll like to challenge Boosters to think about this problem a bit more, and I'd love to see someone who understands the challenges take, say, Boost.System and demonstrate how it could be packaged for either header-only or compiled-library use. The important goal would be to abstract what to done into a general set of guidelines (and any configuration support needed). In other words, we don't so much need a solution for Boost.System as for any Boost library that would benefit from a hybrid computation model.
Comments?
I am keen to explore a hybrid compilation model for Boost.Asio as well. However, some analysis shows that it may be possible to make the Boost.System library header-only while still meeting the goals of short compile times and good software engineering practice. I'll expand on that idea in this email and return to hybrid compilation in a later one. Let's start by assuming that the error_code, error_category and system_error classes in the Boost.System library are updated to match the interfaces described in N2174 [1]. This eliminates the need for a global vector to track error_category registration. My reading of POSIX suggests that it is legitimate for an application to declare a POSIX-supplied function rather than include a header file [2]. The strerror and strerror_r functions can be explicitly declared to avoid including <string.h>. On the other hand, Windows has an excellent track record in maintaining binary compatibility. In my opinion, it is feasible to declare the necessary Windows API functions and constants and thus eliminate the dependency on <windows.h>. There is precedent within boost for this approach [3]. We can use the properties of the extern "C" linkage specification to keep the declarations out of the global namespace [4]. For example: namespace boost { namespace detail { namespace posix { extern "C" char* strerror(int); }}} This can be supplemented with unit tests to ensure that the declarations match those in the corresponding system header file. That about covers it, I think. The explicit declaration approach doesn't scale as well to Boost.Asio because it needs system-defined structures (e.g. sockaddr_in) in addition to function declarations, but that is where a hybrid approach may be useful. Cheers, Chris --- [1] http://www.open-std.org/JTC1/SC22/WG21/docs/papers/2007/n2174.html [2] http://www.opengroup.org/onlinepubs/000095399/xrat/xsh_chap02.html#tag_03_02... [3] boost/detail/interlocked.hpp [4] C++2003, [dcl.link]

On Friday 06 April 2007 16:57, Beman Dawes wrote:
Some users prefer the header-only approach. They need to be able to quickly set up at new project without messing around with compiled libraries. They want to be able to evaluate Boost libraries without doing any preparatory work altogether.
Jeff Garland and Andrey Semashev argued that Boost should adopt a hybrid compilation model that allows applicable Boost libraries to be used in either header-only or compiled-library modes. [...] I'll like to challenge Boosters to think about this problem a bit more, and I'd love to see someone who understands the challenges take, say, Boost.System and demonstrate how it could be packaged for either header-only or compiled-library use.
I wrote a mail describing a way in said thread. This is not a header-only approach though, and it aimed at Boost.Thread not Boost.System. Instead of the header-approach, it simply supplies a single sourcefile that in turn #includes all other necessary sources to compile the library in-place.
The important goal would be to abstract what to done into a general set of guidelines (and any configuration support needed). In other words, we don't so much need a solution for Boost.System as for any Boost library that would benefit from a hybrid computation model.
Guidelines: - Every library that needs separately compiled sources (Thread, System, Regex, ..) should provide an additional file in the include directory named e.g. "compile_in_place.cpp" (I don't care for the name, but it should be the same for all libs). For Thread, that would be boost/thread/compile_in_place.cpp". - This file in turn includes all required sourcefiles for that library. - If several sourcefiles of a library use anonymous namespaces or file-static functions, those need to be checked against conflicts. It might be sufficient to guard those parts with include guards, in particular when they are already in a common file like with Boost.Threads. - Generally, the compilation mode must be configured for static linking (in fact it's just a static library that is compiled 'in-place'). For compilers that support it, autolinking needs to be turned off because there is no .lib file and that would lead to errors. I'm afraid that this needs to be done by the user via a macro, and I think this macro even already exists. - If functionality can't be achieved (IIRC thread-specific storage without DLLs was believed to be impossible), it should be documented as unavailable. Optionally, we could try to cause compile-errors when using such stuff, but that requires that we know that this-or-that lib is 'compiled in-place', which again requires the user to set some macro. Until now, I'd just say "sorry" and leave it at linker errors and documentation. Note: this means that the same source will be compiled in a plethora of different environments, and not just the one explicitly setup by Boost. Ideally, it shouldn't matter. Practically, it will have to cope with e.g. NOMINMAX not being defined on win32 and similar things. Additional checks might be necessary to make the code resistant to such things, but it is generally feasible. I would then also simply generate an error if some preconditions are not met by the environment. In return you solve all problems that are caused by compilers generating differing ABIs, which is IMHO much nastier to diagnose. Uli

Hello, Beman! You wrote on Fri, 06 Apr 2007 10:57:06 -0400: BD> I'll like to challenge Boosters to think about this problem a bit more, BD> and I'd love to see someone who understands the challenges take, say, BD> Boost.System and demonstrate how it could be packaged for either BD> header-only or compiled-library use. The important goal would be to BD> abstract what to done into a general set of guidelines (and any BD> configuration support needed). In other words, we don't so much need a BD> solution for Boost.System as for any Boost library that would benefit BD> from a hybrid computation model. Don't you see that the root of all these problems derived from statement "There are no one standart way do download, deploy, build procedure in C++ world"? If boosters define such procedure and make all these things easy to do, than all, developers and users, will be happy. No need to think about complex configs to support hybrid models or other magic things. Developers will focus on library implementation, not on "how to make this header only?". Users will build apps without spending dozen of hours to figure out how to integrate libxxx into environment. I think we need infrastructure that removes all these issues. If we would have boost.build (I mean v2) widely supported, than having tools that allow smooth and easy IDEs integration(generation vcprojec, bpr etc) than adding new library into existing project will be as easy as "just include this header". This improve build time for both, user and developers. This will allow use system dependent things without worrying to break something. Sorry about my english :(((( With best regards, Konstantin Litvinenko.

Hello, Beman! You wrote on Fri, 06 Apr 2007 10:57:06 -0400: reposting...... BD> I'll like to challenge Boosters to think about this problem a bit more, BD> and I'd love to see someone who understands the challenges take, say, BD> Boost.System and demonstrate how it could be packaged for either BD> header-only or compiled-library use. The important goal would be to BD> abstract what to done into a general set of guidelines (and any BD> configuration support needed). In other words, we don't so much need a BD> solution for Boost.System as for any Boost library that would benefit BD> from a hybrid computation model. Don't you see that the root of all these problems derived from statement "There are no one standart way do download, deploy, build procedure in C++ world"? If boosters define such procedure and make all these things easy to do, than all, developers and users, will be happy. No need to think about complex configs to support hybrid models or other magic things. Developers will focus on library implementation, not on "how to make this header only?". Users will build apps without spending dozen of hours to figure out how to integrate libxxx into environment. I think we need infrastructure that removes all these issues. If we would have boost.build (I mean v2) widely supported, than having tools that allow smooth and easy IDEs integration(generation vcprojec, bpr etc) than adding new library into existing project will be as easy as "just include this header". This improve build time for both, user and developers. This will allow use system dependent things without worrying to break something. Sorry about my english :(((( With best regards, Konstantin Litvinenko.

-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Konstantin Litvinenko Sent: 10 April 2007 12:20 To: boost@lists.boost.org Subject: Re: [boost] Hybrid compilation model? was: [system]
BD> I'll like to challenge Boosters to think about this problem a bit more, BD> and I'd love to see someone who understands the challenges take, say, BD> Boost.System and demonstrate how it could be packaged for either BD> header-only or compiled-library use. The important goal would be to BD> abstract what to done into a general set of guidelines (and any BD> configuration support needed). In other words, we don't so much need a BD> solution for Boost.System as for any Boost library that would benefit BD> from a hybrid computation model.
Don't you see that the root of all these problems derived from statement "There are no one standart way do download, deploy, build procedure in C++ world"? If boosters define such procedure and make all these things easy to do, than all, developers and users, will be happy. No need to think about complex configs to support hybrid models or other magic things. Developers will focus on library implementation, not on "how to make this header only?". Users will build apps without spending dozen of hours to figure out how to integrate libxxx into environment.
I think we need infrastructure that removes all these issues.
Sorry, but I don't think this infrastructure exists, nor will it ever exist. Nor will a Boost 'standard' way of building things become widely adopted. I agree with Beman that we need to cater for both library and header-only. It's more trouble for the library authors, but we should assume they are more expert ;-) Paul --- Paul A Bristow Prizet Farmhouse, Kendal, Cumbria UK LA8 8AB +44 1539561830 & SMS, Mobile +44 7714 330204 & SMS pbristow@hetp.u-net.com
participants (6)
-
Beman Dawes
-
Christopher Kohlhoff
-
Jeff Garland
-
Konstantin Litvinenko
-
Paul A Bristow
-
Ulrich Eckhardt