[system] Header-only Boost.System by default ?
Dear Boost, Is there any reason why Boost system couldn't be set to be header only by default ? Defining both macros allows it : - BOOST_SYSTEM_NO_DEPRECATED - BOOST_ERROR_CODE_HEADER_ONLY This ease, speeds up and reduces binary size greatly, that it's sad that non-aware Boost User needs to link to some dll/so/dylib of Boost.System per default. It has been in many releases of Boost, wouldn't it be good to set it to be the default for a next release ? Cheers, -- Damien Buhl Software Developer +33 6 77 43 10 05
Is there any reason why Boost system couldn't be set to be header only by default ?
Defining both macros allows it :
- BOOST_SYSTEM_NO_DEPRECATED - BOOST_ERROR_CODE_HEADER_ONLY
This ease, speeds up and reduces binary size greatly, that it's sad that non-aware Boost User needs to link to some dll/so/dylib of Boost.System per default.
It has been in many releases of Boost, wouldn't it be good to set it to be the default for a next release ?
The only correct and safe way to use error categories is from a shared library. In header only mode, multiple instances may appear and thus no longer be proper singletons. Stuff breaks in this situation, badly. The above macros are provided for those who don't care if error categories work correctly, or have taken additional measures to ensure correctness. Turning it on by default would lead to users making broken code without realising. It's a build config option for *advanced* users only, ones who know what they're doing. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On Mon, Oct 9, 2017 at 10:34 AM, Niall Douglas via Boost <boost@lists.boost.org> wrote:
The only correct and safe way to use error categories is from a shared library. In header only mode, multiple instances may appear and thus no longer be proper singletons. Stuff breaks in this situation, badly.
If this is true, and I hope that it is not, then we better do something about it and soon, because forcing header-only libraries that want to provide their own error categories to put them in a shared library is a non-starter. Thanks
On 10/10/2017 00:15, Vinnie Falco via Boost wrote:
On Mon, Oct 9, 2017 at 10:34 AM, Niall Douglas via Boost <boost@lists.boost.org> wrote:
The only correct and safe way to use error categories is from a shared library. In header only mode, multiple instances may appear and thus no longer be proper singletons. Stuff breaks in this situation, badly.
If this is true, and I hope that it is not, then we better do something about it and soon, because forcing header-only libraries that want to provide their own error categories to put them in a shared library is a non-starter.
Nothing we can do. WG21 is still thinking about how to solve it properly. All attempts until now to fix the base cause of it have run into quicksand. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On 10/10/2017 12:31, Niall Douglas wrote:
On 10/10/2017 00:15, Vinnie Falco wrote:
On Mon, Oct 9, 2017 at 10:34 AM, Niall Douglas wrote:
The only correct and safe way to use error categories is from a shared library. In header only mode, multiple instances may appear and thus no longer be proper singletons. Stuff breaks in this situation, badly.
If this is true, and I hope that it is not, then we better do something about it and soon, because forcing header-only libraries that want to provide their own error categories to put them in a shared library is a non-starter.
Nothing we can do. WG21 is still thinking about how to solve it properly. All attempts until now to fix the base cause of it have run into quicksand.
Short of magic compiler changes (and possibly OS changes, since it would require having a symbol from multiple shared libraries loaded at the same address), probably the only way to resolve that is to not require singletons -- which means that code that does pointer comparisons on categories has to be hunted down and replaced with code that compares some internal id that is common to all intended-to-be-identical categories, which in turn requires that someone explicitly define a suitably unique id for each custom category. Which is a breaking API contract change and a performance reduction and adds the problem of managing unique category ids. (And ensures hilarity will ensue if someone doesn't use a unique id for different categories.) The other way to prevent the problem is to guarantee that error codes are never passed across shared object boundaries, but I think we can probably agree that in most cases that's a non-starter. Although it is something that header-only libraries are perhaps slightly more likely to get away with (by then pushing the problem to the application author).
On 09/10/2017 19:34, Niall Douglas via Boost wrote:
Is there any reason why Boost system couldn't be set to be header only by default ? [...] The only correct and safe way to use error categories is from a shared library. In header only mode, multiple instances may appear and thus no longer be proper singletons. Stuff breaks in this situation, badly.
If I implement an error category of my own like this : ```cpp inline const basic_error_category& get_basic_error_category() { static basic_error_category category{}; return category; } static const boost::system::error_category& basic_error_category = get_basic_error_category(); ``` Is there really no guarantee that category always get the same address across TUs ? It has been my understanding of extern inline linkage that it actually is so. The linker keeps only one copy in the final program. In C++17 we could even use the static inline specifier for variables. Cheers, -- Damien Buhl
On 10/10/17 10:18, Damien Buhl via Boost wrote:
On 09/10/2017 19:34, Niall Douglas via Boost wrote:
Is there any reason why Boost system couldn't be set to be header only by default ? [...] The only correct and safe way to use error categories is from a shared library. In header only mode, multiple instances may appear and thus no longer be proper singletons. Stuff breaks in this situation, badly.
If I implement an error category of my own like this :
```cpp
inline const basic_error_category& get_basic_error_category() { static basic_error_category category{}; return category; }
static const boost::system::error_category& basic_error_category = get_basic_error_category();
```
Is there really no guarantee that category always get the same address across TUs ?
It will have the same address as long as get_basic_error_category() is called within the same module (dll, so, exe, etc.) If this function is compiled in different modules then there will be multiple instances of the category with different addresses. On Linux and probably other Unix-like systems this can be solved by exporting the category instance, which is the default. I don't think this can be done on Windows - you'd have to export get_basic_error_category for that instead and it will make this code less friendly to header-only libraries.
It has been my understanding of extern inline linkage that it actually is so. The linker keeps only one copy in the final program. In C++17 we could even use the static inline specifier for variables.
I don't think inline variables will solve this problem.
-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Andrey Semashev via Boost Sent: Dienstag, 10. Oktober 2017 09:32 Subject: Re: [boost] [system] Header-only Boost.System by default ?
It will have the same address as long as get_basic_error_category() is called within the same module (dll, so, exe, etc.) If this function is compiled in different modules then there will be multiple instances of the category with different addresses.
On Linux and probably other Unix-like systems this can be solved by exporting the category instance, which is the default. I don't think this can be done on Windows - you'd have to export get_basic_error_category for that instead and it will make this code less friendly to header-only libraries.
And even if the "same address" thing was solvable (which I think it is, although not pretty and it would require interface changes), there is the problem with unloading DLLs/SOs. You'd have to pin every module that contains error categories. Which IMO is a side effect that you really don't expect and want, especially from a header-only library. Paul Groke
How does libstdc++ (>= C++11) solve these problems? Am 10.10.2017 um 10:21 schrieb Groke, Paul via Boost:
-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Andrey Semashev via Boost Sent: Dienstag, 10. Oktober 2017 09:32 Subject: Re: [boost] [system] Header-only Boost.System by default ?
It will have the same address as long as get_basic_error_category() is called within the same module (dll, so, exe, etc.) If this function is compiled in different modules then there will be multiple instances of the category with different addresses.
On Linux and probably other Unix-like systems this can be solved by exporting the category instance, which is the default. I don't think this can be done on Windows - you'd have to export get_basic_error_category for that instead and it will make this code less friendly to header-only libraries. And even if the "same address" thing was solvable (which I think it is, although not pretty and it would require interface changes), there is the problem with unloading DLLs/SOs. You'd have to pin every module that contains error categories. Which IMO is a side effect that you really don't expect and want, especially from a header-only library.
Paul Groke
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 10/10/17 22:27, Mike Gresens via Boost wrote:
Am 10.10.2017 um 10:21 schrieb Groke, Paul via Boost:
-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Andrey Semashev via Boost Sent: Dienstag, 10. Oktober 2017 09:32 Subject: Re: [boost] [system] Header-only Boost.System by default ?
It will have the same address as long as get_basic_error_category() is called within the same module (dll, so, exe, etc.) If this function is compiled in different modules then there will be multiple instances of the category with different addresses.
On Linux and probably other Unix-like systems this can be solved by exporting the category instance, which is the default. I don't think this can be done on Windows - you'd have to export get_basic_error_category for that instead and it will make this code less friendly to header-only libraries. And even if the "same address" thing was solvable (which I think it is, although not pretty and it would require interface changes), there is the problem with unloading DLLs/SOs. You'd have to pin every module that contains error categories. Which IMO is a side effect that you really don't expect and want, especially from a header-only library.
How does libstdc++ (>= C++11) solve these problems?
libstdc++ is not header-only, you link with libstdc++.so.
So, what would happen if boost::system uses aliases to std if we have a modern >= C++11 compiler?! We would have no dependency to libboost_system.so So it's like a header only library. Linking to libstdc++ (or libc++, ...) is always necessary - so no issue. That would be really cool. E.g. using boost::beast with boost::asio, but no runtime dependency to boost libraries. Mike... Am 10.10.2017 um 21:30 schrieb Andrey Semashev via Boost:
On 10/10/17 22:27, Mike Gresens via Boost wrote:
Am 10.10.2017 um 10:21 schrieb Groke, Paul via Boost:
-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Andrey Semashev via Boost Sent: Dienstag, 10. Oktober 2017 09:32 Subject: Re: [boost] [system] Header-only Boost.System by default ?
It will have the same address as long as get_basic_error_category() is called within the same module (dll, so, exe, etc.) If this function is compiled in different modules then there will be multiple instances of the category with different addresses.
On Linux and probably other Unix-like systems this can be solved by exporting the category instance, which is the default. I don't think this can be done on Windows - you'd have to export get_basic_error_category for that instead and it will make this code less friendly to header-only libraries. And even if the "same address" thing was solvable (which I think it is, although not pretty and it would require interface changes), there is the problem with unloading DLLs/SOs. You'd have to pin every module that contains error categories. Which IMO is a side effect that you really don't expect and want, especially from a header-only library.
How does libstdc++ (>= C++11) solve these problems?
libstdc++ is not header-only, you link with libstdc++.so.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 10/10/2017 20:40, Mike Gresens via Boost wrote:
So, what would happen if boost::system uses aliases to std if we have a modern >= C++11 compiler?!
We would have no dependency to libboost_system.so
So it's like a header only library.
Linking to libstdc++ (or libc++, ...) is always necessary - so no issue.
Libraries can and do link to libstdc++.a, not libstdc++.so. That causes multiple std error categories to appear there also. Additionally, one can load a library linked to libstdc++.so via RTLD_LOCAL, so a second copy of libstdc++.so appears, complete with a duplicated set of std error categories. You're probably beginning to see why WG21 hasn't fixed this yet. All of these are problems in systems not under the control of WG21. The only proper way to fix them is for WG21 to embrace the nuclear bomb of implementing its own binary executable format and dynamic loader. The Modules TS is like step 1 of a 20 step program to reach that end goal. Incidentally, MSVC did fix this in VS2015. MSVC is the only platform whose std error categories work right in all circumstances. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On 10/10/2017 09:32, Andrey Semashev via Boost wrote:
[...]
Is there really no guarantee that category always get the same address across TUs ?
It will have the same address as long as get_basic_error_category() is called within the same module (dll, so, exe, etc.) If this function is compiled in different modules then there will be multiple instances of the category with different addresses.
On Linux and probably other Unix-like systems this can be solved by exporting the category instance, which is the default. I don't think this can be done on Windows - you'd have to export get_basic_error_category for that instead and it will make this code less friendly to header-only libraries.
Thanks for these explanations. Is there any unit-tests or are any open program relying on the uniqueness of error_category across modules ? Am 10.10.2017 um 10:21 schrieb Groke, Paul via Boost:
[...] And even if the "same address" thing was solvable (which I think it is, although not pretty and it would require interface changes), there is the problem with unloading DLLs/SOs. You'd have to pin every module that contains error categories. Which IMO is a side effect that you really don't expect and want, especially from a header-only library.
I must be missing something about internal linkage. This problem should already exists with the current implementation no ? If you have an error_category singleton instance in a TU, then unloading a dll containing this TU should also lead to having the error_category refer to some incorrect memory location then. Do you know of any example code doing this ? -- Damien Buhl Software Developer +33 6 77 43 10 05
Is there any unit-tests or are any open program relying on the uniqueness of error_category across modules ?
Yes, all error condition comparisons to error code stops working right if error categories do not have canonical unique addresses. That affects ASIO and Filesystem at the very least.
And even if the "same address" thing was solvable (which I think it is, although not pretty and it would require interface changes), there is the problem with unloading DLLs/SOs. You'd have to pin every module that contains error categories. Which IMO is a side effect that you really don't expect and want, especially from a header-only library.
I must be missing something about internal linkage. This problem should already exists with the current implementation no ? If you have an error_category singleton instance in a TU, then unloading a dll containing this TU should also lead to having the error_category refer to some incorrect memory location then.
Do you know of any example code doing this ?
Python loads its modules with RTLD_LOCAL, so multiple copies of Boost, the STL etc appear in memory. Lots of other programming languages do the same. ELF is really very broken indeed. MachO and PE deficiencies can at least be worked around. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Damien Buhl via Boost Sent: Dienstag, 10. Oktober 2017 21:54 Am 10.10.2017 um 10:21 schrieb Groke, Paul via Boost:
[...] And even if the "same address" thing was solvable (which I think it is, although not pretty and it would require interface changes), there is the problem with unloading DLLs/SOs. You'd have to pin every module that contains error categories. Which IMO is a side effect that you really don't expect and want, especially from a header-only library.
I must be missing something about internal linkage. This problem should already exists with the current implementation no ? If you have an error_category singleton instance in a TU, then unloading a dll containing this TU should also lead to having the error_category refer to some incorrect memory location then.
You're right. It should work though if it's guaranteed that the module that created an error_code object is still loaded while there's a chance that the object may be accessed. If you add some magic that manages to find an already existing equivalent error_category instance, to solve the "more than one instance" problem, an error_code created in module A might end up referencing the error_category object in module B. Then unloading module B would break error_code objects created by A, which really isn't good.
Do you know of any example code doing this ?
Doing what exactly?
On 12/10/2017 10:51, Groke, Paul via Boost wrote: [...]
I must be missing something about internal linkage. This problem should already exists with the current implementation no ? If you have an error_category singleton instance in a TU, then unloading a dll containing this TU should also lead to having the error_category refer to some incorrect memory location then. You're right. It should work though if it's guaranteed that the module that created an error_code object is still loaded while there's a chance that the object may be accessed. If you add some magic that manages to find an already existing equivalent error_category instance, to solve the "more than one instance" problem, an error_code created in module A might end up referencing the error_category object in module B. Then unloading module B would break error_code objects created by A, which really isn't good.
Do you know of any example code doing this ? Doing what exactly? Representing this case where 2 modules link to the same error_category which actually end up being instantiated twice. So that I could play with it. But I can I build it from scratch as well.
What this discussion shows IMHO opinion is that via a dll or via an header only exported static inline function we aren't really able to prevent the library user ending up having more that a single address for the same error categories. There are for both case combinations where this might break, but that's IMO a problem of people builds setup. And it would then really be a problem when the errors begin to be evaluated across their originating modules, which narrows even more the risk that this actually becomes a real problem. So personally I stay inclined that defining BOOST_ERROR_CODE_HEADER_ONLY+BOOST_SYSTEM_NO_DEPRECATED would be a better default in the future. :D Cheers, -- Damien Buhl Software Developer +33 6 77 43 10 05
So personally I stay inclined that defining BOOST_ERROR_CODE_HEADER_ONLY+BOOST_SYSTEM_NO_DEPRECATED would be a better default in the future. :D
Requiring use from a DLL wholly prevents misoperation on Windows and Mac, unless you mix Boost versions. That's why those macros are off by default and should stay that way. (Incidentally SG14 may be proposing a std2::error_code fixing the known problems with std::error_code, but it's some years out yet) Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On Sun, 15 Oct 2017, Niall Douglas via Boost wrote:
So personally I stay inclined that defining BOOST_ERROR_CODE_HEADER_ONLY+BOOST_SYSTEM_NO_DEPRECATED would be a better default in the future. :D
Requiring use from a DLL wholly prevents misoperation on Windows and Mac, unless you mix Boost versions. That's why those macros are off by default and should stay that way.
It would be great to avoid talking about the 2 macros at the same time, since they do not do the same thing. BOOST_SYSTEM_NO_DEPRECATED mostly removes some old synonyms, which otherwise cause #include <boost/system/error_code.hpp> int main(){} to require linking with boost_system. This happens to a lot of users, where the file is included indirectly through other boost packages. If it really has the same dangers as BOOST_ERROR_CODE_HEADER_ONLY (are you sure about that? Is it about boost::throws?), then it should be split into 2 macros, where one only removes the old synonyms and can be enabled by default. -- Marc Glisse
Marc Glisse wrote:
It would be great to avoid talking about the 2 macros at the same time, since they do not do the same thing. BOOST_SYSTEM_NO_DEPRECATED mostly removes some old synonyms, which otherwise cause
#include <boost/system/error_code.hpp> int main(){}
to require linking with boost_system. This happens to a lot of users, where the file is included indirectly through other boost packages.
Perhaps someone could prepare a pull request that turns `#ifndef BOOST_SYSTEM_NO_DEPRECATED` into `#ifdef BOOST_SYSTEM_ENABLE_DEPRECATED`?
On Mon, 16 Oct 2017, Peter Dimov via Boost wrote:
Marc Glisse wrote:
It would be great to avoid talking about the 2 macros at the same time, since they do not do the same thing. BOOST_SYSTEM_NO_DEPRECATED mostly removes some old synonyms, which otherwise cause
#include <boost/system/error_code.hpp> int main(){}
to require linking with boost_system. This happens to a lot of users, where the file is included indirectly through other boost packages.
Perhaps someone could prepare a pull request that turns `#ifndef BOOST_SYSTEM_NO_DEPRECATED` into `#ifdef BOOST_SYSTEM_ENABLE_DEPRECATED`?
Well, Niall seemed to imply that this would be wrong, so it needs to be clarified first. Otherwise: #ifndef BOOST_SYSTEM_ENABLE_DEPRECATED #define BOOST_SYSTEM_NO_DEPRECATED #endif + the doc change. There is a file in chrono that uses BOOST_SYSTEM_NO_DEPRECATED, which restricts the ways to do it if we want to avoid a second pull request in a different package. -- Marc Glisse
Marc Glisse wrote:
Otherwise:
#ifndef BOOST_SYSTEM_ENABLE_DEPRECATED #define BOOST_SYSTEM_NO_DEPRECATED #endif
No, that's not right. Headers included before this one will not see the macro, those after will. Bad things might happen. :-)
There is a file in chrono that uses BOOST_SYSTEM_NO_DEPRECATED...
There is, although it doesn't need to. The non-deprecated path works in either case.
Le 16/10/2017 à 00:33, Peter Dimov via Boost a écrit :
Marc Glisse wrote:
Otherwise:
#ifndef BOOST_SYSTEM_ENABLE_DEPRECATED #define BOOST_SYSTEM_NO_DEPRECATED #endif
No, that's not right. Headers included before this one will not see the macro, those after will. Bad things might happen. :-)
There is a file in chrono that uses BOOST_SYSTEM_NO_DEPRECATED...
There is, although it doesn't need to. The non-deprecated path works in either case. Chrono is using it on the build system and I guess that it shouldn't.
I'll remove it and see what happens. Vicente
Vicente J. Botet Escriba wrote:
Le 16/10/2017 à 00:33, Peter Dimov via Boost a écrit :
Marc Glisse wrote: ...
There is a file in chrono that uses BOOST_SYSTEM_NO_DEPRECATED...
There is, although it doesn't need to. The non-deprecated path works in either case.
Chrono is using it on the build system and I guess that it shouldn't.
I'll remove it and see what happens.
No, the above was in reference to the `#ifdef BOOST_SYSTEM_NO_DEPRECATED ` which the PR https://github.com/boostorg/chrono/pull/29 removed.
Perhaps someone could prepare a pull request that turns `#ifndef BOOST_SYSTEM_NO_DEPRECATED` into `#ifdef BOOST_SYSTEM_ENABLE_DEPRECATED`?
Someone did: https://github.com/boostorg/system/pull/18 We don't have much time left in which to decide whether to move forward with this change in 1.66, so if anyone has an opinion about it, now is the time to speak up.
On Mon, 23 Oct 2017, Peter Dimov via Boost wrote:
Perhaps someone could prepare a pull request that turns `#ifndef BOOST_SYSTEM_NO_DEPRECATED` into `#ifdef BOOST_SYSTEM_ENABLE_DEPRECATED`?
Someone did:
https://github.com/boostorg/system/pull/18
We don't have much time left in which to decide whether to move forward with this change in 1.66, so if anyone has an opinion about it, now is the time to speak up.
Thanks a lot for handling this, the changes look good to me. -- Marc Glisse
On Mon, 9 Oct 2017, Niall Douglas via Boost wrote:
Is there any reason why Boost system couldn't be set to be header only by default ?
Defining both macros allows it :
- BOOST_SYSTEM_NO_DEPRECATED - BOOST_ERROR_CODE_HEADER_ONLY
This ease, speeds up and reduces binary size greatly, that it's sad that non-aware Boost User needs to link to some dll/so/dylib of Boost.System per default.
It has been in many releases of Boost, wouldn't it be good to set it to be the default for a next release ?
The only correct and safe way to use error categories is from a shared library. In header only mode, multiple instances may appear and thus no longer be proper singletons. Stuff breaks in this situation, badly.
The above macros are provided for those who don't care if error categories work correctly, or have taken additional measures to ensure correctness. Turning it on by default would lead to users making broken code without realising. It's a build config option for *advanced* users only, ones who know what they're doing.
Your explanation seems to apply to BOOST_ERROR_CODE_HEADER_ONLY. BOOST_SYSTEM_NO_DEPRECATED is something completely different. As its name implies, it removes some old misfeatures. Misfeatures that force people to link with boost system even when they do not use it... And *not* using error categories is something I can safely do without a shared library. -- Marc Glisse
participants (10)
-
Andrey Semashev -
Damien Buhl -
Gavin Lambert -
Groke, Paul -
Marc Glisse -
Mike Gresens -
Niall Douglas -
Peter Dimov -
Vicente J. Botet Escriba -
Vinnie Falco