[log] [asio] Conflicting default configs
Hi, In Boost.Log I'm using Boost.ASIO to implement network-based syslog backend. Boost.ASIO is used in the compiled part of Boost.Log and, obviously, can be used in the user's code as well. By default Boost.ASIO is header-only, so we basically have a copy of Boost.ASIO in Boost.Log and another copy in the user's app. In Boost.Log I have the option to enable compiler-based TLS at the library build stage (BOOST_LOG_USE_COMPILER_TLS macro), which is by default not enabled. In Boost.ASIO there is a similar option (BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION macro) but the feature is enabled by default. I want Boost.Log built libraries to actually respect the BOOST_LOG_USE_COMPILER_TLS option, so I define BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION internally when BOOST_LOG_USE_COMPILER_TLS is not defined. The problem is that Boost.ASIO changes implementation in an incompatible way when BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION is defined. It is further worsened by a recent change that makes all of Boost.ASIO symbols externally visible. Basically, this means hard to debug trouble on user's end, when different versions of Boost.ASIO (one from Boost.Log binaries and the other used by the user's code directly) are used at the same time. I'd like to ask community what is the preferred resolution of this situation (and possibly similar ones in the future). I can see several ways of tackling this: 1. Introduce an inline ABI namespace in Boost.ASIO, much like the one I have in Boost.Log. The namespace name would depend on BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION and other macros that affect binary compatibility. This would be my preferred solution, although I'm not sure how fast and easy it can be implemented. 2. Remove symbol visibility enforcement from Boost.ASIO, or at least make it optional. That way I could encapsulate my private copy of Boost.ASIO in Boost.Log binaries. 3. Change the defaults, either in Boost.Log or in Boost.ASIO, so that they are aligned by default. When the user changes Boost.Log config he would be expected to know that Boost.ASIO config also needs to be changed accordingly. If he doesn't, Boost.Log is still built consistently with its own config. Note, however, that since Boost.Log is a built library and in most distributions it is built with default options, the choice for Boost.Log is more significant and because of that should probably be more conservative. 4. Same as 3 but also remove the enforcement of Boost.ASIO config by Boost.Log. Whenever the user wants to change the default, he would have to define macros for the both libraries, otherwise Boost.Log can be built inconsistently. 5. Do not change the defaults but remove the config enforcement. BOOST_LOG_USE_COMPILER_TLS becomes Boost.Log-local and as such does not guarantee that the resulting binaries do not in fact use compiler-based TLS. Users can get surprised. 6. Do not change anything, let the users suffer because of ODR issues. Maybe document this somewhere... Opinions?
On 31 Mar 2015 at 1:54, Andrey Semashev wrote:
1. Introduce an inline ABI namespace in Boost.ASIO, much like the one I have in Boost.Log. The namespace name would depend on BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION and other macros that affect binary compatibility. This would be my preferred solution, although I'm not sure how fast and easy it can be implemented.
Why don't you just use standalone ASIO internally? It has a different ABI, and I believe is expected to not interact with Boost.ASIO. The internal copy can be generated using Chris's special "include all of ASIO" magic file. Just fire it through a bit of python which implements only the #include directive or use a STL excluding preprocessor to generate a single file including all of standalone ASIO.
2. Remove symbol visibility enforcement from Boost.ASIO, or at least make it optional. That way I could encapsulate my private copy of Boost.ASIO in Boost.Log binaries.
You'd still run into ABI mismatch if the end user uses a different Boost ASIO version to you which is not uncommon (users like to pin the version of Boost and Boost.ASIO separately). The only valid visibility of an internal ASIO is hidden, but then you have to be real careful you never allow Log to be used as anything but a shared object else you'll cause all ASIOs to be hidden too as hidden visibility is sticky. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On 31/03/2015 13:41, Niall Douglas wrote:
On 31 Mar 2015 at 1:54, Andrey Semashev wrote:
1. Introduce an inline ABI namespace in Boost.ASIO, much like the one I have in Boost.Log. The namespace name would depend on BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION and other macros that affect binary compatibility. This would be my preferred solution, although I'm not sure how fast and easy it can be implemented.
Why don't you just use standalone ASIO internally? It has a different ABI, and I believe is expected to not interact with Boost.ASIO.
The internal copy can be generated using Chris's special "include all of ASIO" magic file. Just fire it through a bit of python which implements only the #include directive or use a STL excluding preprocessor to generate a single file including all of standalone ASIO.
If you can tweak the namespace it uses to be something inside boost::log rather than whatever it defaults to, that should help avoid collisions between Boost.Log and either Boost.Asio or the user using standalone ASIO themselves.
On Tuesday 31 March 2015 20:05:42 Gavin Lambert wrote:
On 31/03/2015 13:41, Niall Douglas wrote:
On 31 Mar 2015 at 1:54, Andrey Semashev wrote:
1. Introduce an inline ABI namespace in Boost.ASIO, much like the one I have in Boost.Log. The namespace name would depend on BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION and other macros that affect binary compatibility. This would be my preferred solution, although I'm not sure how fast and easy it can be implemented.
Why don't you just use standalone ASIO internally? It has a different ABI, and I believe is expected to not interact with Boost.ASIO.
The internal copy can be generated using Chris's special "include all of ASIO" magic file. Just fire it through a bit of python which implements only the #include directive or use a STL excluding preprocessor to generate a single file including all of standalone ASIO.
If you can tweak the namespace it uses to be something inside boost::log rather than whatever it defaults to, that should help avoid collisions between Boost.Log and either Boost.Asio or the user using standalone ASIO themselves.
Hmm, this made me thinking if I could #include Boost.ASIO headers inside a namespace. Sounds hackish and certainly not the expected use case by ASIO developers, but it could work.
On Tuesday 31 March 2015 11:38:45 you wrote:
On Tuesday 31 March 2015 20:05:42 Gavin Lambert wrote:
On 31/03/2015 13:41, Niall Douglas wrote:
On 31 Mar 2015 at 1:54, Andrey Semashev wrote:
1. Introduce an inline ABI namespace in Boost.ASIO, much like the one I have in Boost.Log. The namespace name would depend on BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION and other macros that affect binary compatibility. This would be my preferred solution, although I'm not sure how fast and easy it can be implemented.
Why don't you just use standalone ASIO internally? It has a different ABI, and I believe is expected to not interact with Boost.ASIO.
The internal copy can be generated using Chris's special "include all of ASIO" magic file. Just fire it through a bit of python which implements only the #include directive or use a STL excluding preprocessor to generate a single file including all of standalone ASIO.
If you can tweak the namespace it uses to be something inside boost::log rather than whatever it defaults to, that should help avoid collisions between Boost.Log and either Boost.Asio or the user using standalone ASIO themselves.
Hmm, this made me thinking if I could #include Boost.ASIO headers inside a namespace. Sounds hackish and certainly not the expected use case by ASIO developers, but it could work.
On the second thought, no, this won't work because other headers, including Boost, STL and system headers, will be included under my namespace and this will certainly break at some point.
On 31 Mar 2015 at 11:38, Andrey Semashev wrote:
If you can tweak the namespace it uses to be something inside boost::log rather than whatever it defaults to, that should help avoid collisions between Boost.Log and either Boost.Asio or the user using standalone ASIO themselves.
Hmm, this made me thinking if I could #include Boost.ASIO headers inside a namespace. Sounds hackish and certainly not the expected use case by ASIO developers, but it could work.
Boost.ASIO is auto generated by a script from standalone ASIO. Some of the things it does is to wrap the asio namespace with a boost namespace, the ASIO_ macros with a BOOST_ prefix and so on. Tweaking that script to prepend a log_internal namespace and a BOOST_LOG_INTERAL_ instead should be trivial. I'd have your build system optionally git clone the standalone ASIO repo and optionally refresh a locally kept copy. BTW Boost.ASIO lags standalone ASIO by up to a year. They are not ABI compatible. Indeed right now standalone ASIO has deprecated null_buffers, so any code using null_buffers (surely a lot) is considered legacy. Or put another way, the Networking TS won't have null_buffers. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On Tuesday 31 March 2015 13:30:03 Niall Douglas wrote:
Boost.ASIO is auto generated by a script from standalone ASIO. Some of the things it does is to wrap the asio namespace with a boost namespace, the ASIO_ macros with a BOOST_ prefix and so on.
Tweaking that script to prepend a log_internal namespace and a BOOST_LOG_INTERAL_ instead should be trivial. I'd have your build system optionally git clone the standalone ASIO repo and optionally refresh a locally kept copy.
Yes, I understand, and this is what I referred to as "copy&paste" in my previous reply. Taking this route would mean I would have to maintain this copy of ASIO on my own, possibly refreshing it from time to time, let alone the code bloat. Given that I only need to send UDP packets and resolve domain names once in a while, I don't think the effort is worthwhile. What I'd like to see is a solution for Boost.ASIO, as well as other header- only libraries that support ABI-breaking config options. Ideally, there should be a guideline for future libraries. As for Boost.Log case, well, I'm still uncertain, but leaning towards dropping Boost.ASIO and writing my own network compatibility layer.
BTW Boost.ASIO lags standalone ASIO by up to a year.
This is very unfortunate.
On 31 Mar 2015 at 20:38, Andrey Semashev wrote:
What I'd like to see is a solution for Boost.ASIO, as well as other header- only libraries that support ABI-breaking config options. Ideally, there should be a guideline for future libraries.
That's exactly BindLib. A python script it supplies will stamp out the ABI versioning preprocessor metaprogramming for all the config options which affect ABI. The preprocessor metaprogramming then assembles the appropriate inline namespacing. The clang AST parser program spits out the appropriate template aliasing. It all comes together surprisingly well. I have a unit test for AFIO which configures two extremely ABI incompatible copies of AFIO (one entirely STL11 based, the other entirely Boost based) in the same translation unit and it executes both sets of unit tests. Takes about seven minutes to compile, but all unit tests pass green on all platforms.
As for Boost.Log case, well, I'm still uncertain, but leaning towards dropping Boost.ASIO and writing my own network compatibility layer.
Before doing that I'd search for a header only lightweight C++ (or C) sockets platform abstraction library. I believe there is more than one available.
BTW Boost.ASIO lags standalone ASIO by up to a year.
This is very unfortunate.
Boost.ASIO is the ultra stable edition. Bug fixes are cherry picked into Boost.ASIO where easy to do so. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On Tuesday 31 March 2015 01:41:56 Niall Douglas wrote:
On 31 Mar 2015 at 1:54, Andrey Semashev wrote:
1. Introduce an inline ABI namespace in Boost.ASIO, much like the one I have in Boost.Log. The namespace name would depend on BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION and other macros that affect binary compatibility. This would be my preferred solution, although I'm not sure how fast and easy it can be implemented.
Why don't you just use standalone ASIO internally? It has a different ABI, and I believe is expected to not interact with Boost.ASIO.
That would be an external dependency, which may not be present at build time. And it doesn't fundamentally fix the problem - it would still cause trouble if the user uses a differently configured standalone ASIO.
The internal copy can be generated using Chris's special "include all of ASIO" magic file. Just fire it through a bit of python which implements only the #include directive or use a STL excluding preprocessor to generate a single file including all of standalone ASIO.
Oh, you mean to basically copy&paste ASIO into Boost.Log under some different namespace? I guess that's possible, although I don't need that much of ASIO and that would certainly be a bloat. It might be better to remove Boost.ASIO and use native API directly. I would like to avoid this approach if another is possible.
2. Remove symbol visibility enforcement from Boost.ASIO, or at least make it optional. That way I could encapsulate my private copy of Boost.ASIO in Boost.Log binaries.
You'd still run into ABI mismatch if the end user uses a different Boost ASIO version to you which is not uncommon (users like to pin the version of Boost and Boost.ASIO separately). The only valid visibility of an internal ASIO is hidden, but then you have to be real careful you never allow Log to be used as anything but a shared object else you'll cause all ASIOs to be hidden too as hidden visibility is sticky.
If different Boost.ASIO versions (or different configurations of Boost.ASIO) are used and Boost.Log is a static lib then yes, that option wouldn't help. Actually, I'm not sure if different releases of Boost.ASIO are workable at all in this setup. I guess, the solution in this case should involve name mangling (e.g. the internal namespace).
AMDG On 03/30/2015 04:54 PM, Andrey Semashev wrote:
1. Introduce an inline ABI namespace in Boost.ASIO, much like the one I have in Boost.Log. The namespace name would depend on BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION and other macros that affect binary compatibility. This would be my preferred solution, although I'm not sure how fast and easy it can be implemented.
2. Remove symbol visibility enforcement from Boost.ASIO, or at least make it optional. That way I could encapsulate my private copy of Boost.ASIO in Boost.Log binaries.
Both of these will tend to result in pointlessly duplicating ASIO.
3. Change the defaults, either in Boost.Log or in Boost.ASIO, so that they are aligned by default. When the user changes Boost.Log config he would be expected to know that Boost.ASIO config also needs to be changed accordingly. If he doesn't, Boost.Log is still built consistently with its own config. Note, however, that since Boost.Log is a built library and in most distributions it is built with default options, the choice for Boost.Log is more significant and because of that should probably be more conservative.
4. Same as 3 but also remove the enforcement of Boost.ASIO config by Boost.Log. Whenever the user wants to change the default, he would have to define macros for the both libraries, otherwise Boost.Log can be built inconsistently.
5. Do not change the defaults but remove the config enforcement. BOOST_LOG_USE_COMPILER_TLS becomes Boost.Log-local and as such does not guarantee that the resulting binaries do not in fact use compiler-based TLS. Users can get surprised.
I don't think that libraries should set configuration options for other libraries.
6. Do not change anything, let the users suffer because of ODR issues. Maybe document this somewhere...
In Christ, Steven Watanabe
On Tuesday 31 March 2015 09:14:30 Steven Watanabe wrote:
AMDG
On 03/30/2015 04:54 PM, Andrey Semashev wrote:
1. Introduce an inline ABI namespace in Boost.ASIO, much like the one I have in Boost.Log. The namespace name would depend on BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION and other macros that affect binary compatibility. This would be my preferred solution, although I'm not sure how fast and easy it can be implemented.
2. Remove symbol visibility enforcement from Boost.ASIO, or at least make it optional. That way I could encapsulate my private copy of Boost.ASIO in Boost.Log binaries.
Both of these will tend to result in pointlessly duplicating ASIO.
Well, Boost.ASIO code is currently compiled into Boost.Log, so unless the user links statically the duplication already exists.
I don't think that libraries should set configuration options for other libraries.
What if the default config of the other library is not suitable? The reason why I added config enforcement is because compiler-based TLS can prevent using Boost.Log in dynamically loaded (i.e. with dlopen()) libraries. So with default config Boost.Log binaries are inconsistent and not working in some cases.
What if the default config of the other library is not suitable? The reason why I added config enforcement is because compiler-based TLS can prevent using Boost.Log in dynamically loaded (i.e. with dlopen()) libraries. So with default config Boost.Log binaries are inconsistent and not working in some cases.
In my opinion, this is up to the user of the library to decide. Duplicating the implementation of an entire library to work around different configs is one reason some libraries get a bad reputation, please don't do it. Instead, the user should configure the libraries as appropriate to match the features they need. If the user needs to use Boost.Log in shared objects and ASIO is not able to do that by default, then that is opt-in and, in my opinion, should be explicit to ensure they link against an appropriate ASIO. It seems to me that Log depends on ASIO so therefore Log needs to respect the ASIO default by detecting BOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION and setting any internal flags appropriately. -- chris
participants (5)
-
Andrey Semashev -
Chris Glover -
Gavin Lambert -
Niall Douglas -
Steven Watanabe