[testing] Add a tester with hidden visibility
Hi, Is it possible to add a tester that enables hidden visibility (- fvisibility=hidden -fvisibility-inlines-hidden) when building and testing Boost? I believe, a single tester on Linux+GCC would be enough to discover most issues related to symbol visibility. The reason I'm asking is that I believe Boost should support building with hidden visibility by default. This mode allows more optimizations and does not export unintended symbols from libraries. I've set hidden visibility for Boost.Log (in develop) and, as I understand, Robert is doing a similar change in Boost.Serialization. However, when building 1.58 with hidden visibility for my project I found out that not all libraries work well with that setting. What do you think about it?
I've recently had experience with just this question while maintaining the serialization library. The serialization library can be built as a DLL. The windows version builds with MSVC only exporting those symbols which are required from outside. That is, it builds with the MSVC of visibility=hidden. I set visibility=hidden in the jam file and built with gcc. The library then failed to link with most of the tests. So I had to update the source code to support the attributes for symbol export on those functions which need to be exported. It sounds simple, but getting this right in a way that supports both gcc and MSVC is tricky and hard to get correct. I got the gcc to build and pass all test, but now the MSVC is broken. Unfortunately, our testing system truncates the error message so I can't see why the current branch in develop fails to build. I'm still waiting from a response from the boost build team. In the course of addressing this, I figured - how hard can this be and decided to see how other libraries do it. Checking the jam files of other libraries - there are only a few which actually build as DLLS, I found that no one actually builds with visibility=hidden. I would be very surprised if any of them actually build correctly with this switch set. To do this, it has to be done for each individual library. You can't just add the switch and re-build. Robert Ramey -- View this message in context: http://boost.2283326.n4.nabble.com/testing-Add-a-tester-with-hidden-visibili... Sent from the Boost - Dev mailing list archive at Nabble.com.
On Thursday 07 May 2015 07:54:14 Robert Ramey wrote:
I've recently had experience with just this question while maintaining the serialization library.
The serialization library can be built as a DLL. The windows version builds with MSVC only exporting those symbols which are required from outside. That is, it builds with the MSVC of visibility=hidden.
I set visibility=hidden in the jam file and built with gcc. The library then failed to link with most of the tests. So I had to update the source code to support the attributes for symbol export on those functions which need to be exported. It sounds simple, but getting this right in a way that supports both gcc and MSVC is tricky and hard to get correct. I got the gcc to build and pass all test, but now the MSVC is broken. Unfortunately, our testing system truncates the error message so I can't see why the current branch in develop fails to build. I'm still waiting from a response from the boost build team.
In the course of addressing this, I figured - how hard can this be and decided to see how other libraries do it. Checking the jam files of other libraries - there are only a few which actually build as DLLS, I found that no one actually builds with visibility=hidden. I would be very surprised if any of them actually build correctly with this switch set. To do this, it has to be done for each individual library. You can't just add the switch and re-build.
I'm not sure I understand your point. Unless we don't plan to ever support hidden visibility, we at least need to start testing this configuration.
The original question suggested that if a tester changed the jam file so that the switch gcc visibility=hidden was enabled then tests would pass. My point is that his is very unlikely which is why it isn't being done. If visibility=hidden is to be supported (and I think it should be) library source code will have to be updated. Robert Ramey -- View this message in context: http://boost.2283326.n4.nabble.com/testing-Add-a-tester-with-hidden-visibili... Sent from the Boost - Dev mailing list archive at Nabble.com.
On Thursday 07 May 2015 09:22:26 Robert Ramey wrote:
The original question suggested that if a tester changed the jam file so that the switch gcc visibility=hidden was enabled then tests would pass.
I did not suggest that the tests would immediately pass. But I would like to have such tester - for users to see which libraries have problems, and for library maintainers to be able to fix those problems.
My point is that his is very unlikely which is why it isn't being done.
I'm trying to build 1.58 with visibility=hidden right now and so far the only library that's failing to build is Wave. I didn't run tests yet, but the start looks promising to me.
If visibility=hidden is to be supported (and I think it should be) library source code will have to be updated.
Sure.
Andrey Semashev-2 wrote
On Thursday 07 May 2015 09:22:26 Robert Ramey wrote:
The original question suggested that if a tester changed the jam file so that the switch gcc visibility=hidden was enabled then tests would pass.
I did not suggest that the tests would immediately pass. But I would like to have such tester - for users to see which libraries have problems, and for library maintainers to be able to fix those problems.
If the jam file includes the visibility=hidden switch then ALL the testers will be using it by default. So rather than approaching the testers, convince the library developers to update their jamfiles. Robert Ramey -- View this message in context: http://boost.2283326.n4.nabble.com/testing-Add-a-tester-with-hidden-visibili... Sent from the Boost - Dev mailing list archive at Nabble.com.
On Friday 08 May 2015 08:21:58 Robert Ramey wrote:
Andrey Semashev-2 wrote
On Thursday 07 May 2015 09:22:26 Robert Ramey wrote:
The original question suggested that if a tester changed the jam file so that the switch gcc visibility=hidden was enabled then tests would pass.
I did not suggest that the tests would immediately pass. But I would like to have such tester - for users to see which libraries have problems, and for library maintainers to be able to fix those problems.
If the jam file includes the visibility=hidden switch then ALL the testers will be using it by default.
No, the flag can be added in user-config.jam, on the tester's site.
Andrey Semashev-2 wrote
On Friday 08 May 2015 08:21:58 Robert Ramey wrote:
Andrey Semashev-2 wrote
The original question suggested that if a tester changed the jam file so that the switch gcc visibility=hidden was enabled then tests would
On Thursday 07 May 2015 09:22:26 Robert Ramey wrote: pass.
I did not suggest that the tests would immediately pass. But I would like to have such tester - for users to see which libraries have problems, and for library maintainers to be able to fix those problems.
If the jam file includes the visibility=hidden switch then ALL the testers will be using it by default.
No, the flag can be added in user-config.jam, on the tester's site.
yes - if the jam file includes the flag, then ALL testers will be using it by default. I don't think this is disputable. Furthermore, once one makes any required changes to support gcc visibility=hidden I don't think there is any scenario for which one would not want to use it. So the best solution would be to added to the jam file whenever it's supported. Robert Ramey. -- View this message in context: http://boost.2283326.n4.nabble.com/testing-Add-a-tester-with-hidden-visibili... Sent from the Boost - Dev mailing list archive at Nabble.com.
On Friday 08 May 2015 15:49:39 Robert Ramey wrote:
Andrey Semashev-2 wrote
On Friday 08 May 2015 08:21:58 Robert Ramey wrote:
If the jam file includes the visibility=hidden switch then ALL the testers will be using it by default.
No, the flag can be added in user-config.jam, on the tester's site.
yes - if the jam file includes the flag, then ALL testers will be using it by default.
How a flag added for a single toolset on a single tester would affect all testers?
Furthermore, once one makes any required changes to support gcc visibility=hidden I don't think there is any scenario for which one would not want to use it. So the best solution would be to added to the jam file whenever it's supported.
I'm sorry, you lost me here.
Andrey Semashev-2 wrote
On Friday 08 May 2015 15:49:39 Robert Ramey wrote:
Andrey Semashev-2 wrote
On Friday 08 May 2015 08:21:58 Robert Ramey wrote:
If the jam file includes the visibility=hidden switch then ALL the testers will be using it by default.
No, the flag can be added in user-config.jam, on the tester's site.
yes - if the jam file includes the flag, then ALL testers will be using it by default.
How a flag added for a single toolset on a single tester would affect all testers?
It wouldn't. My view is that : if a library can be built and used with visibility=hidden there is no reason not to build it this way, this flag should be in the Jamfile. If a library cannot be built and used with visibility=hidden there is no reason to build it this way so if visibility=hidden isn't in the Jamfile, adding for a tester is a pointless exercise. Hence, visibility=hidden should be added to the Jamfile if possible so that the library will be built and tested with visibility=hidden if and only if it's supported. There is no scenario under which adding it to project.jam would be beneficial. Robert Ramey -- View this message in context: http://boost.2283326.n4.nabble.com/testing-Add-a-tester-with-hidden-visibili... Sent from the Boost - Dev mailing list archive at Nabble.com.
The original question suggested that if a tester changed the jam file so that the switch gcc visibility=hidden was enabled then tests would pass. My point is that his is very unlikely which is why it isn't being done. If visibility=hidden is to be supported (and I think it should be) library source code will have to be updated.
It's a simplistic case, but the Math/TR1 library is built with -fvisibility-hidden with both GCC and Intel C++ John.
The original question suggested that if a tester changed the jam file so that the switch gcc visibility=hidden was enabled then tests would pass. My point is that his is very unlikely which is why it isn't being done. If visibility=hidden is to be supported (and I think it should be) library source code will have to be updated.
I think it should just work - just ran the regex lib test suite on cygwin with -fvisibility=hidden and everything worked just fine. John.
On 7 May 2015 at 16:40, Andrey Semashev wrote:
Is it possible to add a tester that enables hidden visibility (- fvisibility=hidden -fvisibility-inlines-hidden) when building and testing Boost? I believe, a single tester on Linux+GCC would be enough to discover most issues related to symbol visibility.
The reason I'm asking is that I believe Boost should support building with hidden visibility by default. This mode allows more optimizations and does not export unintended symbols from libraries.
I've set hidden visibility for Boost.Log (in develop) and, as I understand, Robert is doing a similar change in Boost.Serialization. However, when building 1.58 with hidden visibility for my project I found out that not all libraries work well with that setting.
What do you think about it?
Many years ago when Dave Abrahams asked me to contribute visibility support to Boost he hoped to persuade all library maintainers to add support and for all libraries to be built with default hidden visibility such that ELF shared objects gained some of the reuse stability of PE DLLs. At the time such a move was controversial and considered retrograde by a sufficient proportion of C++ experts, and so that effort never panned out. Dave, at that time, was irritated by this attitude if I remember correctly. Now everyone is much more comfortable with the idea of actively managing shared library coupling at the binary level, I think there is a chance that flipping hidden visibility on for all Boost.Build and forcing maintainers to finally do the right thing in their libraries is very worth considering. After all, most libraries now support Windows DLLs, and if you already support Windows DLLs it's a very small addition to support ELF symbol export. Just follow the instructions at https://gcc.gnu.org/wiki/Visibility using this as a template: #if (defined(BOOST_AFIO_DYN_LINK) || defined(BOOST_ALL_DYN_LINK)) && \ !defined(BOOST_AFIO_STATIC_LINK) # if defined(BOOST_AFIO_SOURCE) # undef BOOST_AFIO_HEADERS_ONLY # define BOOST_AFIO_DECL BOOST_SYMBOL_EXPORT # define BOOST_AFIO_BUILD_DLL # else # define BOOST_AFIO_DECL # endif #else # define BOOST_AFIO_DECL #endif // building a shared library ... and then markup all public types and functions with a BOOST_AFIO_DECL. And make sure when building your library BOOST_AFIO_SOURCE is defined. Done. On 7 May 2015 at 7:54, Robert Ramey wrote:
So I had to update the source code to support the attributes for symbol export on those functions which need to be exported. It sounds simple, but getting this right in a way that supports both gcc and MSVC is tricky and hard to get correct. I got the gcc to build and pass all test, but now the MSVC is broken. Unfortunately, our testing system truncates the error message so I can't see why the current branch in develop fails to build. I'm still waiting from a response from the boost build team.
I have no idea why you continue to insist on trying to second guess the very well established pattern here. The design of the GCC visibility support was chosen back in 2004 to correctly support all combinations of GCC, MSVC and Mingw without any special casing, trickery, or any macro work above the traditional preprocessor logic used for MSVC. One macro system for marking up export/import. Same system for all compilers. Easy, if you make it easy on yourself. Just reuse, without addition, what works on MSVC to build DLLs. Mark up exception throwable types with BOOST_SYMBOL_VISIBLE so clang works. Done. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On Thursday 07 May 2015 16:45:22 Niall Douglas wrote:
On 7 May 2015 at 7:54, Robert Ramey wrote:
So I had to update the source code to support the attributes for symbol export on those functions which need to be exported. It sounds simple, but getting this right in a way that supports both gcc and MSVC is tricky and hard to get correct. I got the gcc to build and pass all test, but now the MSVC is broken. Unfortunately, our testing system truncates the error message so I can't see why the current branch in develop fails to build. I'm still waiting from a response from the boost build team.
I have no idea why you continue to insist on trying to second guess the very well established pattern here. The design of the GCC visibility support was chosen back in 2004 to correctly support all combinations of GCC, MSVC and Mingw without any special casing, trickery, or any macro work above the traditional preprocessor logic used for MSVC.
Well, it's not that simple. For example, if you dllexport a class template and explicitly instantiate it in a dll with a non-exported type as a template parameter, you will get what you want - an exported specialization. Hidden visibility, OTOH, is viral, so the instantiated template will silently become hidden in this case. Other than that, yes, it's rather similar to MSVC.
On 7 May 2015 at 18:59, Andrey Semashev wrote:
I have no idea why you continue to insist on trying to second guess the very well established pattern here. The design of the GCC visibility support was chosen back in 2004 to correctly support all combinations of GCC, MSVC and Mingw without any special casing, trickery, or any macro work above the traditional preprocessor logic used for MSVC.
Well, it's not that simple. For example, if you dllexport a class template and explicitly instantiate it in a dll with a non-exported type as a template parameter, you will get what you want - an exported specialization. Hidden visibility, OTOH, is viral, so the instantiated template will silently become hidden in this case.
Other than that, yes, it's rather similar to MSVC.
It is a bad idea to dllexport templates as it greatly increases the chance of an ODR violation biting you in the ass (e.g. two internal private types with the same name). Nor is it ever necessary, as template implementation is always header only. Just leave templates, or any header implemented code, alone. Let them be compiled into a local implementation for every ELF object. Remove anything you really need to have exactly a single implementation of into a non-templated base class which is dllexported. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On Thursday 07 May 2015 17:32:13 Niall Douglas wrote:
On 7 May 2015 at 18:59, Andrey Semashev wrote:
I have no idea why you continue to insist on trying to second guess the very well established pattern here. The design of the GCC visibility support was chosen back in 2004 to correctly support all combinations of GCC, MSVC and Mingw without any special casing, trickery, or any macro work above the traditional preprocessor logic used for MSVC.
Well, it's not that simple. For example, if you dllexport a class template and explicitly instantiate it in a dll with a non-exported type as a template parameter, you will get what you want - an exported specialization. Hidden visibility, OTOH, is viral, so the instantiated template will silently become hidden in this case.
Other than that, yes, it's rather similar to MSVC.
It is a bad idea to dllexport templates as it greatly increases the chance of an ODR violation biting you in the ass (e.g. two internal private types with the same name). Nor is it ever necessary, as template implementation is always header only. Just leave templates, or any header implemented code, alone. Let them be compiled into a local implementation for every ELF object. Remove anything you really need to have exactly a single implementation of into a non-templated base class which is dllexported.
That's a very simplistic approach. In practice it is often needed to export a template specialization for different reasons: dependency hiding, compilation speedup, code size reduction, etc. Of course, there are complications and caveats but "just don't do it" is often not an answer.
On 7 May 2015 at 19:39, Andrey Semashev wrote:
It is a bad idea to dllexport templates as it greatly increases the chance of an ODR violation biting you in the ass (e.g. two internal private types with the same name). Nor is it ever necessary, as template implementation is always header only. Just leave templates, or any header implemented code, alone. Let them be compiled into a local implementation for every ELF object. Remove anything you really need to have exactly a single implementation of into a non-templated base class which is dllexported.
That's a very simplistic approach. In practice it is often needed to export a template specialization for different reasons: dependency hiding, compilation speedup, code size reduction, etc. Of course, there are complications and caveats but "just don't do it" is often not an answer.
Up until C++ 11 I really would say "just don't do it". The compiler doesn't know not to instantiate dllexported templates and does so anyway, so compilation speedup is zero. Marking templates with dllexport means the linker must deal with a ton more symbols to match up and remove because they aren't hidden, so you lose on link times too. The only potential win is code size reduction, and given that most compilers compile template instantiations into two variants, inlined at point of use and extern, the latter rarely actually gets used and is therefore thrown away by the linker anyway, so the gains are usually not worth the significant added hassle and brittleness for most libraries with most reasonably sized templates. Since C++ 11 we have extern templates, and that finally makes dllexporting template instantiations useful. I can see very significant potential benefits especially to compile and link times there, but a much superior implementation is to mark all common instantiations extern template in the headers and to supply a separate standalone library (without default hidden visibility) with all the common instantiations in one place. So you would have a two library solution there, one with with default hidden with the normal code and common template instantiations marked extern, and a second with default visible with the common template instantiations supplied. I hope that made sense. My point is, trying to do portable visibility management for templates from within a single library is likely much harder, much slower, and more brittle than a two library solution. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
Up until C++ 11 I really would say "just don't do it". The compiler doesn't know not to instantiate dllexported templates and does so anyway, so compilation speedup is zero. Marking templates with dllexport means the linker must deal with a ton more symbols to match up and remove because they aren't hidden, so you lose on link times too. The only potential win is code size reduction, and given that most compilers compile template instantiations into two variants, inlined at point of use and extern, the latter rarely actually gets used and is therefore thrown away by the linker anyway, so the gains are usually not worth the significant added hassle and brittleness for most libraries with most reasonably sized templates.
Since C++ 11 we have extern templates, and that finally makes dllexporting template instantiations useful. I can see very significant potential benefits especially to compile and link times there,
Boost.Regex has done this since day 1 for compilers that support it (msvc and gcc). Compile time speedups are larger on msvc, but in both cases well worth the effort. In fact it's one reason the library hasn't gone header-only: it could probably be done, but compile times are just *so* much faster when the template instances are in an external lib... John.
On 7 May 2015 at 18:38, John Maddock wrote:
Since C++ 11 we have extern templates, and that finally makes dllexporting template instantiations useful. I can see very significant potential benefits especially to compile and link times there,
Boost.Regex has done this since day 1 for compilers that support it (msvc and gcc). Compile time speedups are larger on msvc, but in both cases well worth the effort. In fact it's one reason the library hasn't gone header-only: it could probably be done, but compile times are just *so* much faster when the template instances are in an external lib...
Good stuff. For reference, AFIO a very simple library sees this with precompiled headers enabled: GCC: 12m => 10m MSVC: 7m => 2m clang: No change AFIO also has the ability to both use precompiled headers and not be header only: GCC: 4m MSVC: 1m So especially big wins for GCC. I suspect that C++ Modules cannot deliver any improvements over the non-header only with precompiled headers configuration. Good build config always beats compiler magic. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
For reference, AFIO a very simple library sees this with precompiled headers enabled:
GCC: 12m => 10m MSVC: 7m => 2m clang: No change
AFIO also has the ability to both use precompiled headers and not be header only:
GCC: 4m MSVC: 1m
So especially big wins for GCC.
I suspect that C++ Modules cannot deliver any improvements over the non-header only with precompiled headers configuration. Good build config always beats compiler magic.
Actually it can - if implemented well - way back when, Borland's C++ compiler had better PCH support than anything else out there, it wasn't quite C++ modules, but it was quite a decent part of the way there. They had a little server app that managed and loaded all required PCH's as AST's into memory, so when a file was being compiled it would just map the memory from the other process and start compiling from there. It was lightning fast with near enough instant compile times. And this was 10+ years ago on hardware that folks would laugh at today. It was more or less automatic too - if there were no changes to a files #includes it would just go ahead and cache everything till after the last #include without the user having to excplicitly mess with PCH's at all. Of course both templates and C++ in general are a lot more complex now: this was all pre-STL days! Still it would be nice, if we could have a compiler/IDE release that wasn't actually 3 times slower than the one before ;) With rose tinted spectacles yours, John.
On 7 May 2015 at 19:10, John Maddock wrote:
I suspect that C++ Modules cannot deliver any improvements over the non-header only with precompiled headers configuration. Good build config always beats compiler magic.
Actually it can - if implemented well - way back when, Borland's C++ compiler had better PCH support than anything else out there, it wasn't quite C++ modules, but it was quite a decent part of the way there. They had a little server app that managed and loaded all required PCH's as AST's into memory, so when a file was being compiled it would just map the memory from the other process and start compiling from there. It was lightning fast with near enough instant compile times. And this was 10+ years ago on hardware that folks would laugh at today. It was more or less automatic too - if there were no changes to a files #includes it would just go ahead and cache everything till after the last #include without the user having to excplicitly mess with PCH's at all. Of course both templates and C++ in general are a lot more complex now: this was all pre-STL days! Still it would be nice, if we could have a compiler/IDE release that wasn't actually 3 times slower than the one before ;)
It looks like I accidentally deleted a subsequent paragraph about C++ Modules being more likely useful as a step towards next gen tooling :( My C++ Now 2014 white paper (http://arxiv.org/abs/1405.3323) detailed how one might implement a C++ compiler as a database of ASTs such that all C++ was always as if inline defined i.e. every bit of code could see all other C++ code anywhere on the system. Build and dependency management simply become graph database queries. I think one of the C++ Now 2015 talks might cover some actual work done on such a database repository of ASTs to implement a build and package manager (this one? http://sched.co/2oLN), it's ultimately where Link Time Optimisation must eventually end up anyway when pushed to extremis. Might as well get a JIT zero time build system for free out of it. I guess the question eventually becomes if the compiler vendors are game for implementing such a very complex tool. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
participants (4)
-
Andrey Semashev -
John Maddock -
Niall Douglas -
Robert Ramey