[serialization] Errors when testing withe mingw-64/gcc on Windows
When running serialization tests with mingw-64/gcc on Windows I get repeated errors of the form: ..\..\../boost/archive/detail/basic_oarchive.hpp:64:5: error: function 'boost::archive::detail::helper_collection& boost::archive::detail::basic_oarchive::get_helper_collection()' definition is marked dllimport get_helper_collection(){ Is this a known problem ? Testing with VC++ does not give any errors.
On 7/17/15 10:51 PM, Edward Diener wrote:
When running serialization tests with mingw-64/gcc on Windows I get repeated errors of the form:
..\..\../boost/archive/detail/basic_oarchive.hpp:64:5: error: function 'boost::archive::detail::helper_collection& boost::archive::detail::basic_oarchive::get_helper_collection()' definition is marked dllimport get_helper_collection(){
Is this a known problem ?
Testing with VC++ does not give any errors.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
A couple of months ago I made efforts to limit symbol visibility on compilers other than Visual Studio. This was to resolve complaints about the serialization library exporting many more symbols than necessary on these platforms. This turned out to be quite a bit more intricate than I expected as there's some confusion between vs and other platforms on this point and they use different syntax. I managed to get it all together - except for MinGW - which still has problems in this area. I noticed it was failing the MinGW tests (when they were being run). But I couldn't really fix it because I have neither Visual Studio nor MinGW. As I said, it turned out some what more complex than I expected due to one thing in particular. The wide version of the DLL exports it's own symbols, but import symbols from the the base version of the DLL. So one has to get it just right. For a while I was experiencing the situation where fixing it for one version would create problems in the other. I never could figure out which syntax MinGW uses. If one has all three (or four) platforms gcc, mingw, clang, and Visual studion and some patience it's an easy fix - once you discover it. But without this setup, it's sort of chasing one's tail. So if you want to do this, I would be grateful. Not that if you get it wrong, it shows up in some tests but not in others. So one has to be prepared to run the whole serialization test suite before declaring victory. Robert Ramey
On 7/18/2015 10:04 AM, Robert Ramey wrote:
On 7/17/15 10:51 PM, Edward Diener wrote:
When running serialization tests with mingw-64/gcc on Windows I get repeated errors of the form:
..\..\../boost/archive/detail/basic_oarchive.hpp:64:5: error: function 'boost::archive::detail::helper_collection& boost::archive::detail::basic_oarchive::get_helper_collection()' definition is marked dllimport get_helper_collection(){
Is this a known problem ?
Testing with VC++ does not give any errors.
A couple of months ago I made efforts to limit symbol visibility on compilers other than Visual Studio. This was to resolve complaints about the serialization library exporting many more symbols than necessary on these platforms. This turned out to be quite a bit more intricate than I expected as there's some confusion between vs and other platforms on this point and they use different syntax. I managed to get it all together - except for MinGW - which still has problems in this area. I noticed it was failing the MinGW tests (when they were being run). But I couldn't really fix it because I have neither Visual Studio nor MinGW.
As I said, it turned out some what more complex than I expected due to one thing in particular. The wide version of the DLL exports it's own symbols, but import symbols from the the base version of the DLL. So one has to get it just right. For a while I was experiencing the situation where fixing it for one version would create problems in the other. I never could figure out which syntax MinGW uses. If one has all three (or four) platforms gcc, mingw, clang, and Visual studion and some patience it's an easy fix - once you discover it. But without this setup, it's sort of chasing one's tail.
So if you want to do this, I would be grateful. Not that if you get it wrong, it shows up in some tests but not in others. So one has to be prepared to run the whole serialization test suite before declaring victory.
I will look at it. I can run the serialization tests with mingw/mingw-64/gcc, clang, and VC++ on Windows and gcc and clang on Linux.
On 7/18/2015 10:04 AM, Robert Ramey wrote:
On 7/17/15 10:51 PM, Edward Diener wrote:
When running serialization tests with mingw-64/gcc on Windows I get repeated errors of the form:
..\..\../boost/archive/detail/basic_oarchive.hpp:64:5: error: function 'boost::archive::detail::helper_collection& boost::archive::detail::basic_oarchive::get_helper_collection()' definition is marked dllimport get_helper_collection(){
Is this a known problem ?
Testing with VC++ does not give any errors.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
A couple of months ago I made efforts to limit symbol visibility on compilers other than Visual Studio. This was to resolve complaints about the serialization library exporting many more symbols than necessary on these platforms. This turned out to be quite a bit more intricate than I expected as there's some confusion between vs and other platforms on this point and they use different syntax. I managed to get it all together - except for MinGW - which still has problems in this area. I noticed it was failing the MinGW tests (when they were being run). But I couldn't really fix it because I have neither Visual Studio nor MinGW.
As I said, it turned out some what more complex than I expected due to one thing in particular. The wide version of the DLL exports it's own symbols, but import symbols from the the base version of the DLL. So one has to get it just right. For a while I was experiencing the situation where fixing it for one version would create problems in the other. I never could figure out which syntax MinGW uses. If one has all three (or four) platforms gcc, mingw, clang, and Visual studion and some patience it's an easy fix - once you discover it. But without this setup, it's sort of chasing one's tail.
So if you want to do this, I would be grateful. Not that if you get it wrong, it shows up in some tests but not in others. So one has to be prepared to run the whole serialization test suite before declaring victory.
I don't know how "to run the whole serialization test suite" but here is what I have done: 1) I have created PR "Changes to fix serialization for gcc and clang on Windows #19". This fixes the building of the serialization library for mingw(-64)/gcc, both regular and wide character. 2) For clang I have been told that the system of exporting/importing individual member functions as opposed to entire class does not work, since clang will not export a class's RTTI unless the entire class is exported. So while the build of the regular serialization library succeeds it is probably not usable from other clang code. This can be seen in the fact that using your current system of exporting/importing individual member functions of a visible class I can not build the wide character serialization library using clang because it refuses to link with the dependent regular character serialization library. So unless you want to change the way that export/import works for clang on Windows clang on Windows will not work with serialization. 3) I have tested my changes with gcc and clang on Linux and with gcc, VC++, and clang on Windows. As explained above everything succeeds except clang on Windows, which also failed before my changes. If you could look at this PR and try it with your "whole serialization test suite" it would be appreciated. If you are willing to give the go ahead I can try changing clang on Windows so that the entire class is exported rather than individual member functions for that one-off case. It is pretty easy to test for clang on Windows using gcc at the preprocessor level. I do realize exporting the entire class in the one-off clang case changes your work in carefully honing visibility, for this single case, but I cannot change the way clang on Windows works. Changing for clang on Windows will not change any other compiler implementation but will make your code slightly messier. PS. I also created PR "Delete recreated files. #18" which deletes some files from Git which were being regenerated by serialization tests each time.
On 8/3/15 11:58 AM, Edward Diener wrote:
I don't know how "to run the whole serialization test suite" but here is what I have done:
It's just running all the tests via the normal boost build/bjam system. I'm in the habit of mentioning this because it's quite common for someone to post a "fix" that is "too trivial" to require testing and it's helpful for me to avoid this.
1) I have created PR "Changes to fix serialization for gcc and clang on Windows #19". This fixes the building of the serialization library for mingw(-64)/gcc, both regular and wide character.
I will consider this shortly. I've been trying to catch up on other stuff so I've got a few things pending with the serialization library maintenance. I'll look at the PR soon.
2) For clang I have been told that the system of exporting/importing individual member functions as opposed to entire class does not work, since clang will not export a class's RTTI unless the entire class is exported.
Hmmm - my problems showed up only when I enabled limited visibility. I managed to get all tests passing on gcc, clang, and msvc (with a little help from my friends). Then I got nailed by mingw which I couldn't figure out the visibility system. So while the build of the regular serialization library
succeeds it is probably not usable from other clang code.
hmmm - all the tests passed on my system with both static and shared library builds. This can be
seen in the fact that using your current system of exporting/importing individual member functions of a visible class I can not build the wide character serialization library using clang because it refuses to link with the dependent regular character serialization library.
Again, I'm not seeing this - I've only had problems with mingw So unless
you want to change the way that export/import works for clang on Windows clang on Windows will not work with serialization.
I can't remember right now whether I originally had been exporting the whole class or just function by function. It's been 6 months. One reason that it's more complicated with the serialization library is that there are two DLLS the core one and the wide character add-in. The second exports functions of it's own while importing functions from the core. This is doable, but confusing.
3) I have tested my changes with gcc and clang on Linux and with gcc, VC++, and clang on Windows. As explained above everything succeeds except clang on Windows, which also failed before my changes.
Hmmm - I'm not sure how this is different than the current status. As I said, my pending issues were with mingw. Since mingw is also a relative of windows - it wouldn't surprise me of clang for windows also has issues. I am interested in getting this sorted out since I want to keep the serialization library as widely applicable as possible and I appreciate your help with this (and other people too). On the other hand, I'm beginning to wonder. The problem with mingw has been there since 1.58 and no one has complained. Same goes for clang on windows. It's hard to get motivated to spend time making things work for platforms which it doesn't seem anyone is using.
If you could look at this PR and try it with your "whole serialization test suite" it would be appreciated.
I absolutely will, and soon. As I said, the whole test suite is just the normal boost build test. One thing that I do is run all the combinations of shared/static/debug/release for gcc and clang - I don't have a windows development system any more. This takes a while sucks up an incredible amount of disk space.
If you are willing to give the go ahead I can try changing clang on Windows so that the entire class is exported rather than individual member functions for that one-off case. It is pretty easy to test for clang on Windows using gcc at the preprocessor level. I do realize exporting the entire class in the one-off clang case changes your work in carefully honing visibility, for this single case, but I cannot change the way clang on Windows works. Changing for clang on Windows will not change any other compiler implementation but will make your code slightly messier.
LOL - you mean the serialization library is messy?
PS. I also created PR "Delete recreated files. #18" which deletes some files from Git which were being regenerated by serialization tests each time.
Fine. Again I much appreciate your efforts. I just need to find the time to give them attention they deserve. Robert Ramey
On 8/3/2015 3:45 PM, Robert Ramey wrote:
On 8/3/15 11:58 AM, Edward Diener wrote:
I don't know how "to run the whole serialization test suite" but here is what I have done:
It's just running all the tests via the normal boost build/bjam system. I'm in the habit of mentioning this because it's quite common for someone to post a "fix" that is "too trivial" to require testing and it's helpful for me to avoid this.
Understood.
1) I have created PR "Changes to fix serialization for gcc and clang on Windows #19". This fixes the building of the serialization library for mingw(-64)/gcc, both regular and wide character.
I will consider this shortly. I've been trying to catch up on other stuff so I've got a few things pending with the serialization library maintenance. I'll look at the PR soon.
2) For clang I have been told that the system of exporting/importing individual member functions as opposed to entire class does not work, since clang will not export a class's RTTI unless the entire class is exported.
Hmmm - my problems showed up only when I enabled limited visibility. I managed to get all tests passing on gcc, clang, and msvc (with a little help from my friends). Then I got nailed by mingw which I couldn't figure out the visibility system.
My PR fixes mingw.
So while the build of the regular serialization library
succeeds it is probably not usable from other clang code.
hmmm - all the tests passed on my system with both static and shared library builds.
Did you yourself test clang on Windows targeting mingw(-64)/gcc ? Which version(s) of clang on Windows did you test ? Did you test the latest from source ? I am not speaking at all about clang on Linux, which works correctly with serialization. If I checkout serialization on 'master', go to the serialization build directory and execute "b2 toolset=clang-3.7" the final result is: clang-linux.link.dll ..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\boost_wserialization-clang37-d-1_59.dll ..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\basic_text_wiprimitive.obj: In function `ZN5boost13serialization15throw_exceptionINS_7archive17archive_exceptionEEEvRKT_': C:\Programming\VersionControl\modular-boost\libs\serialization\build/..\..\..\boost/serialization/throw_exception.hpp:36: undefined reference to `typeinfo for boost::archive::archive_exception' ..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\basic_text_wiprimitive.obj: In function `archive_exception': C:\Programming\VersionControl\modular-boost\libs\serialization\build/..\..\..\boost/archive/archive_exception.hpp:42: undefined reference to `vtable for boost::archive::archive_exception' ..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\text_wiarchive.obj:(.rdata[__ZTIN5boost7archive6detail15common_iarchiveINS0_14text_wiarchiveEEE]+0x10): undefined reference to `typeinfo for boost::archive::detail::basic_iarchive' ..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\text_woarchive.obj:(.rdata[__ZTIN5boost7archive6detail15common_oarchiveINS0_14text_woarchiveEEE]+0x10): undefined reference to `typeinfo for boost::archive::detail::basic_oarchive' ..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wgrammar.obj:(.rdata[__ZTCN5boost7archive21xml_archive_exceptionE4_NS0_17archive_exceptionE]+0x10): undefined reference to `typeinfo for boost::archive::archive_exception' ..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wgrammar.obj:(.rdata[__ZTCN5boost7archive21xml_archive_exceptionE4_NS0_17archive_exceptionE]+0x2c): undefined reference to `typeinfo for boost::archive::archive_exception' ..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wgrammar.obj:(.rdata[__ZTIN5boost7archive21xml_archive_exceptionE]+0x10): undefined reference to `typeinfo for boost::archive::archive_exception' ..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wiarchive.obj:(.rdata[__ZTIN5boost7archive6detail15common_iarchiveINS0_13xml_wiarchiveEEE]+0x10): undefined reference to `typeinfo for boost::archive::detail::basic_iarchive' ..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_woarchive.obj:(.rdata[__ZTIN5boost7archive6detail15common_oarchiveINS0_13xml_woarchiveEEE]+0x10): undefined reference to `typeinfo for boost::archive::detail::basic_oarchive' clang++.exe: error: linker command failed with exit code 1 (use -v to see invocation) clang++.exe: error: linker command failed with exit code 1 (use -v to see invocation) "C:/Programming/VersionControl/bninja_installed_clang/bin/clang++.exe" -o "..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\boost_wserialization-clang37-d-1_59.dll" -Wl,-soname -Wl,boost_wserialization-clang37-d-1_59.dll -shared -Wl,--start-group "..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\basic_text_wiprimitive.obj" "..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\basic_text_woprimitive.obj" "..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\text_wiarchive.obj" "..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\text_woarchive.obj" "..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\utf8_codecvt_facet.obj" "..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wgrammar.obj" "..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wiarchive.obj" "..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_woarchive.obj" "..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\codecvt_null.obj" "..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\boost_serialization-clang37-d-1_59.dll" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -g -march=i686 -m32 ...failed clang-linux.link.dll ..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\boost_wserialization-clang37-d-1_59.dll... ...failed updating 1 target... The same thing happens when I build serialization using clang-3.4.1, clang-3.5.2, and clang-3.6.1. If I switch serialization to 'develop' the result is the same. Notice that the regular serialization build succeeds and it is only the wide character serialization build that fails with the above messages. However the regular serialization library built for clang has some problems ( see below ). If I run the serialization tests using clang ( the latest 3.7 or any of the earlier versions I can run ) I get errors because some things are not really exported/imported as one would assume it should be. Here is the result: ...failed updating 167 targets... ...skipped 676 targets... ...updated 1127 targets...
This can be
seen in the fact that using your current system of exporting/importing individual member functions of a visible class I can not build the wide character serialization library using clang because it refuses to link with the dependent regular character serialization library.
Again, I'm not seeing this - I've only had problems with mingw
So unless
you want to change the way that export/import works for clang on Windows clang on Windows will not work with serialization.
I can't remember right now whether I originally had been exporting the whole class or just function by function. It's been 6 months.
One reason that it's more complicated with the serialization library is that there are two DLLS the core one and the wide character add-in. The second exports functions of it's own while importing functions from the core. This is doable, but confusing.
I can see how it is working and it is understandable to me. There is nothing intrinsically unusual about one shared library depending on another shared library. It certainly should work.
3) I have tested my changes with gcc and clang on Linux and with gcc, VC++, and clang on Windows. As explained above everything succeeds except clang on Windows, which also failed before my changes.
Hmmm - I'm not sure how this is different than the current status. As I said, my pending issues were with mingw. Since mingw is also a relative of windows - it wouldn't surprise me of clang for windows also has issues.
I was able to solve the mingw problems by e-mails on the mingw-64 mailing list. The basic problem was getting mingw-64/gcc to correctly export the RTTI for certain classes. It turns out that for a class with exported virtual functions even certain implicit compiler generated member functions must be explicitly specified and exported also. I could never get the rules for this explained to me but by trial and error I managed to explicitly export the right functions. The two problematic classes in the srialization library were archive_exception and xml_archive_exception. In my PR you will see the changes I had to make to get these two classes to export/import correctly in mingw(-64)/gcc.
I am interested in getting this sorted out since I want to keep the serialization library as widely applicable as possible and I appreciate your help with this (and other people too). On the other hand, I'm beginning to wonder. The problem with mingw has been there since 1.58 and no one has complained. Same goes for clang on windows. It's hard to get motivated to spend time making things work for platforms which it doesn't seem anyone is using.
My PR fixes mingw, but not clang. In my discussion about this on the clang developer mailing list I was told very explicitly that clang is incapable of exporting just specific member functions of certain classes because it can not export the RTTI information for a class unless the class is exported as a whole. The problem seems to occur with classes which are exporting virtual functions. The culprits again in serialization are archive_exception and xml_archive_exception. There are very unfortunately no regression tests run on Windows for anything other than VC++. I have the setup but I just don't have the resources to run the regression tests for various versions of mingw(-64)/gcc and clang/mingw(-64) on Windows.
If you could look at this PR and try it with your "whole serialization test suite" it would be appreciated.
I absolutely will, and soon.
As I said, the whole test suite is just the normal boost build test. One thing that I do is run all the combinations of shared/static/debug/release for gcc and clang - I don't have a windows development system any more. This takes a while sucks up an incredible amount of disk space.
If you are willing to give the go ahead I can try changing clang on Windows so that the entire class is exported rather than individual member functions for that one-off case. It is pretty easy to test for clang on Windows using gcc at the preprocessor level. I do realize exporting the entire class in the one-off clang case changes your work in carefully honing visibility, for this single case, but I cannot change the way clang on Windows works. Changing for clang on Windows will not change any other compiler implementation but will make your code slightly messier.
LOL - you mean the serialization library is messy?
<g> No, of course not. What I of course meant is that a one-off system will make the code a little more difficult to follow as far as the visibility and importing/exporting is concerned. Actually it should not be so bad. I can probably manipulate detail/decl.hpp to do it for clang on Windows.
PS. I also created PR "Delete recreated files. #18" which deletes some files from Git which were being regenerated by serialization tests each time.
Fine.
Again I much appreciate your efforts. I just need to find the time to give them attention they deserve.
participants (2)
-
Edward Diener -
Robert Ramey