[serialization] 1.35 problem with explicit instantiation
With 1.34, I was using the pattern described by Robert (http://lists.boost.org/boost-users/2006/01/16293.php) with non-intrusive serialization. In short, the serialization functions are merely declared in the header files. They are implemented in the .cpp file, and then explicitly instantiated for each archive type I'm using. However, with 1.35, this no longer works. The export macros create an instance of the guid initializer as follows: export.hpp:150 BOOST_PP_CAT(boost_serialization_guid_initializer_, __LINE__) However, this gives me redefinition errors in the following situaion: ClassA.h -------- (line 1:) BOOST_CLASS_EXPORT(ClassA); ClassB.h -------- (line 1: )BOOST_CLASS_EXPORT(ClassB); main.cpp #include <ClassA.h> #include <ClassB.h> (create an archive instance and attempt to serialize) ------------------------------- As you can see, two objects, both named boost_serialization_guid_initializer_1, will be created, and the compiler (MSVC80) will give me a redefinition error. I honestly have no idea what's going on. This is so obvious, it has to mean that I'm organizing my code in some weird fashion. Thanks! Dan
I've updated the advice on using BOOST_CLASS_EXPORT. The new advice is described in the documentation in the trunk. This will require moving some macros between source modules (*.cpp and *.hpp files). In exchange, you'll get a) a more carefully thought out way to handle exports. b) the ability to include serialization code in DLLS with little or no suprises. c) serialization code will be thread safe - without any locking required. Robert Ramey Dan Thill wrote:
With 1.34, I was using the pattern described by Robert (http://lists.boost.org/boost-users/2006/01/16293.php) with non-intrusive serialization. In short, the serialization functions are merely declared in the header files. They are implemented in the .cpp file, and then explicitly instantiated for each archive type I'm using.
However, with 1.35, this no longer works. The export macros create an instance of the guid initializer as follows:
export.hpp:150
BOOST_PP_CAT(boost_serialization_guid_initializer_, __LINE__)
However, this gives me redefinition errors in the following situaion:
ClassA.h -------- (line 1:) BOOST_CLASS_EXPORT(ClassA);
ClassB.h -------- (line 1: )BOOST_CLASS_EXPORT(ClassB);
main.cpp
#include <ClassA.h> #include <ClassB.h>
(create an archive instance and attempt to serialize)
-------------------------------
As you can see, two objects, both named boost_serialization_guid_initializer_1, will be created, and the compiler (MSVC80) will give me a redefinition error.
I honestly have no idea what's going on. This is so obvious, it has to mean that I'm organizing my code in some weird fashion.
Thanks! Dan
I've updated the advice on using BOOST_CLASS_EXPORT. The new advice is described in the documentation in the trunk. This will require moving some macros between source modules (*.cpp and *.hpp files).
In exchange, you'll get
a) a more carefully thought out way to handle exports. b) the ability to include serialization code in DLLS with little or no suprises. c) serialization code will be thread safe - without any locking required.
Robert, I've read the new documentation and while it fixes the compilation errors, I still get runtime unregistered_class exceptions. Here's my general setup, briefly. MyClass.h ------- class MyClass : public SomeBase { .... }; MyClassSZ.h --------- #include <MyClass.h> template<Archive> void serialize( Archive&, MyClass&, ... ) ClassSZ.cpp ----------- #include <MyClassSZ.h> #include <boost/archive/...> // All archive types I'm going to use #include <boost/serialization/export.hpp> BOOST_CLASS_EXPORT(MyClass); void serialize( Archive&, MyClass&, unsigned int ) { // perform serialization. No void_cast_register is done // because I *always* serialize via a base class pointer. // also, I do not serialize the base object as there is // no data in it. } // Explicitly instantiate above function for each archive type template void serialize<xml_iarchive>(xml_iarchive&, MyClass&, unsigned int); template void serialize<xml_oarchive>(xml_iarchive&, MyClass&, unsigned int); // ... etc. ---------------------------------------- All of my ClassSZ.cpp files are compiled and statically linked into a library. Then, in the application: main.cpp -------- #include ... // archive type that I want #include <MyClassSZ.h> // Create archive and serialize a MyClass via SomeBase ptr) ---------------------------------------- Is it a possibility that the linker is not linking in the guid_initializer code? For this particular build type, I'm not even doing optimizations... let alone link-time code generation. This is on MSVC80. Thanks, Dan
Is it a possibility that the linker is not linking in the guid_initializer code? For this particular build type, I'm not even doing optimizations... let alone link-time code generation. This is on MSVC80.
Hm. The MSVC debugger isn't letting me set breakpoints in any of my serialization code, saying "This breakpoint won't be hit. No symbols have been loaded for this document" I know that the serialization code is being generated in the static library (compilation times are lengthy) and that library is specified on the linker commandline. But after that... -dan
Dan Thill wrote:
Is it a possibility that the linker is not linking in the guid_initializer code? For this particular build type, I'm not even doing optimizations... let alone link-time code generation. This is on MSVC80.
Hm. The MSVC debugger isn't letting me set breakpoints in any of my serialization code, saying "This breakpoint won't be hit. No symbols have been loaded for this document"
I know that the serialization code is being generated in the static library (compilation times are lengthy) and that library is specified on the linker commandline. But after that...
To reply to myself once more... Yes, this is the problem. I put a breakpoint at extended_type_info_typeid.hpp:96, and it is not being called for any types which I serialize *only* via base class pointer. For types that are serialized through most-derived pointer (sharing the same base as the others), an extended_type_info_typeid_1<T> is getting created for each one. -Dan
Dan Thill wrote:
Dan Thill wrote:
Is it a possibility that the linker is not linking in the guid_initializer code? For this particular build type, I'm not even doing optimizations... let alone link-time code generation. This is on MSVC80. Hm. The MSVC debugger isn't letting me set breakpoints in any of my serialization code, saying "This breakpoint won't be hit. No symbols have been loaded for this document"
I know that the serialization code is being generated in the static library (compilation times are lengthy) and that library is specified on the linker commandline. But after that...
To reply to myself once more...
Yes, this is the problem. I put a breakpoint at extended_type_info_typeid.hpp:96, and it is not being called for any types which I serialize *only* via base class pointer. For types that are serialized through most-derived pointer (sharing the same base as the others), an extended_type_info_typeid_1<T> is getting created for each one.
Hey Dan, I think the problem is that you are using a static library which contains the serialization code. Static libraries are notorious for not linking code that is never directly called. So the solution is to include the object files contained in your static library into your executable/dll. For Visual C++, there is no command line linker option as I understand it. On gcc, you can do g++ -Wl,--whole-archive if memory serves (which I'm sure it doesn't) -- Sohail Somani http://uint32t.blogspot.com
I think the problem is that you are using a static library which contains the serialization code. Static libraries are notorious for not linking code that is never directly called. So the solution is to include the object files contained in your static library into your executable/dll. For Visual C++, there is no command line linker option as I understand it. On gcc, you can do g++ -Wl,--whole-archive if memory serves (which I'm sure it doesn't)
I'm beginning to see that this is the case. However, there are several references on this list over the years for this exact organization. It worked on 1.34.1 -- I suppose because BOOST_CLASS_EXPORT had to be called in the header file. However, with 1.35, the new system which avoids the header ordering issue, which requires BOOST_CLASS_EXPORT to be in the implementation file, breaks this approach. I thought that was one of the benefits of the new system: to better force instantiation of the static objects which make the registration work. And, according to the trunk version, putting serialization code in DLLs is supposed to work--using the same organization I'm trying to use now. But it looks like it won't work with static libs. I'm sorry if I sound irate. I'm just extremely frustrated. I went through all the effort to put the 1.34.1 serialization into its own library in order to fix compilation time problems in our product. And then, a few days later, 1.35 is released, breaking everything. (and for various platform issues, we need to move to 1.35). -Dan
Dan Thill wrote:
I think the problem is that you are using a static library which contains the serialization code. Static libraries are notorious for not linking code that is never directly called. So the solution is to include the object files contained in your static library into your executable/dll. For Visual C++, there is no command line linker option as I understand it. On gcc, you can do g++ -Wl,--whole-archive if memory serves (which I'm sure it doesn't)
I'm beginning to see that this is the case. However, there are several references on this list over the years for this exact organization. It worked on 1.34.1 -- I suppose because BOOST_CLASS_EXPORT had to be called in the header file. However, with 1.35, the new system which avoids the header ordering issue, which requires BOOST_CLASS_EXPORT to be in the implementation file, breaks this approach. I thought that was one of the benefits of the new system: to better force instantiation of the static objects which make the registration work.
Yes, but if your linker throws away the code, then there is no point.
And, according to the trunk version, putting serialization code in DLLs is supposed to work--using the same organization I'm trying to use now. But it looks like it won't work with static libs.
This is a misfeature of static libraries, not anything else. For example, if you want to have a function thats called when your library is loaded into memory, gcc provides an attribute that looks something like: void call_me_please() __attribute__((constructor)) {...} But if you put that function in a static library, it never gets looked at because it is never referenced. So you *have* to link the object containing call_me_please into the final executable/dll. The same is true for the serialization code, as it works similarly to the above.
I'm sorry if I sound irate. I'm just extremely frustrated. I went through all the effort to put the 1.34.1 serialization into its own library in order to fix compilation time problems in our product. And then, a few days later, 1.35 is released, breaking everything. (and for various platform issues, we need to move to 1.35).
Do you mean compile-time or link-time? Anyway, I think the only reason 1.34 worked for you is because you had BOOST_CLASS_EXPORT in the header files. Otherwise I would have expected the same thing to happen. If you really want a separate library, I think you have to include all the object files in your final link. This helps compile-times, but not link-times. Maybe the incremental linker will work for you. Good luck. -- Sohail Somani http://uint32t.blogspot.com
Yes, but if your linker throws away the code, then there is no point.
Indeed. I thought voodoo to somehow prevent this, however :)
Do you mean compile-time or link-time? Anyway, I think the only reason 1.34 worked for you is because you had BOOST_CLASS_EXPORT in the header files. Otherwise I would have expected the same thing to happen.
Compile time. Particularly, a socket client (using ASIO) which serializes many objects. When working on the communications client, each compile would take forever because of all the template instantiations of the serialized classes. Thanks for helping clear all this up for me. -Dan
Dan Thill:
And, according to the trunk version, putting serialization code in DLLs is supposed to work--using the same organization I'm trying to use now. But it looks like it won't work with static libs.
The easiest way to force the inclusion of a particular .o(bj) file is to call a function from it. You could call a function from each .cpp file in which you have an export macro. This is in fact not much different than just having "register<X>()" calls instead of export macros, which is why I tend to prefer this low-tech approach instead of relying on implicit registration via static object constructors. Maybe the serialization library could provide an alternative function-based registration mechanism that would be immune to the static library problem. Or maybe it already provides it: guid_initializer< T >().export_guid( "T" ); This is less convenient, but has the advantage of working. :-)
Peter Dimov wrote: [snip]
Maybe the serialization library could provide an alternative function-based registration mechanism that would be immune to the static library problem. Or maybe it already provides it:
guid_initializer< T >().export_guid( "T" );
This is less convenient, but has the advantage of working. :-)
As long as ADL works on your compiler. I think for maximum portability, you need to use register_type and void_cast_register. Theoretically anyway. -- Sohail Somani http://uint32t.blogspot.com
Peter Dimov wrote:
Dan Thill:
And, according to the trunk version, putting serialization code in DLLs is supposed to work--using the same organization I'm trying to use now. But it looks like it won't work with static libs.
The easiest way to force the inclusion of a particular .o(bj) file is to call a function from it. You could call a function from each .cpp file in which you have an export macro.
This is in fact not much different than just having "register<X>()" calls instead of export macros, which is why I tend to prefer this low-tech approach instead of relying on implicit registration via static object constructors.
Maybe the serialization library could provide an alternative function-based registration mechanism that would be immune to the static library problem. Or maybe it already provides it:
guid_initializer< T >().export_guid( "T" );
This is less convenient, but has the advantage of working. :-)
I believe that making a *.cpp file with the macros BOOST_CLASS_EXPORT(T) would be exactly equivalent to the above. Robert Ramey
Robert Ramey:
Peter Dimov wrote:
Maybe the serialization library could provide an alternative function-based registration mechanism that would be immune to the static library problem. Or maybe it already provides it:
guid_initializer< T >().export_guid( "T" );
This is less convenient, but has the advantage of working. :-)
I believe that making a *.cpp file with the macros BOOST_CLASS_EXPORT(T) would be exactly equivalent to the above.
Well, it depends on where the .cpp file is. If it's part of the static library, it will not be included by the linker because there are no references to it. But it is true that A.cpp: void register_types() { guid_initializer<T1>().export_guid("T1"); guid_initializer<T2>().export_guid("T2"); guid_initializer<T3>().export_guid("T3"); } and B.cpp: BOOST_CLASS_EXPORT(T1) BOOST_CLASS_EXPORT(T2) BOOST_CLASS_EXPORT(T3) void register_types() { } are more or less equivalent (as part of the static library).
Peter Dimov wrote:
Robert Ramey:
Peter Dimov wrote:
I believe that making a *.cpp file with the macros BOOST_CLASS_EXPORT(T) would be exactly equivalent to the above.
Well, it depends on where the .cpp file is. If it's part of the static library, it will not be included by the linker because there are no references to it. But it is true that
What I meant was that if you want to make sure the stuff in the static library is linked in, just make a small *.cpp file as part of your main program which includes the BOOST_CLASS_EXPORT macros. #include "a.hpp" #include "b.hpp" ... BOOST_CLASS_EXPORT(A) BOOST_CLASS_EXPORT(B) .... Robert Ramey
Dan Thill wrote: I'm certainly sympathetic. But I fee that now we've resolved several big problems which were really annoying. a) The library should now be thread-safe - without any locking ! b) The situation of having serialization code in DLLS - and possibly more than one was never considered when the library was made. I believe this has now been addressed. c) Export was always confusing. Hopefully now it is less so. Also in using the old advice, code was instantiated whether it was used or not. So, I AM sympathic as the annoys the hell out of me when it happens to me. You could make a separt little module which you would compile into your main which would include all the BOOST_CLASS_EXPORT macros for your types. Then the linker would get the code from your static library. Robert Ramey
I'm sorry if I sound irate. I'm just extremely frustrated. I went through all the effort to put the 1.34.1 serialization into its own library in order to fix compilation time problems in our product. And then, a few days later, 1.35 is released, breaking everything. (and for various platform issues, we need to move to 1.35).
-Dan
You could make a separt little module which you would compile into your main which would include all the BOOST_CLASS_EXPORT macros for your types. Then the linker would get the code from your static library.
This sounds intriguing. Say I put all the BOOST_CLASS_EXPORT macros into a single header file. Wouldn't I run into trouble if I included that header file twice in the same executable, due to the multiple definitions of the static guid_initializer. Or would the linker see that they are exactly the same, and only choose one? Or is the answer "depends" :) Thank you. -Dan
Dan Thill wrote:
You could make a separt little module which you would compile into your main which would include all the BOOST_CLASS_EXPORT macros for your types. Then the linker would get the code from your static library.
This sounds intriguing. Say I put all the BOOST_CLASS_EXPORT macros into a single header file. Wouldn't I run into trouble if I included that header file twice in the same executable, due to the multiple definitions of the static guid_initializer. Or would the linker see that they are exactly the same, and only choose one? Or is the answer "depends" :)
Thank you.
-Dan
My suggestion is to include BOOST_CLASS_EXPORT in just one *.cpp file. This "registration" module would be called only to build with static libraries Robert Ramey
participants (5)
-
Dan Thill -
Nat Goodspeed -
Peter Dimov -
Robert Ramey -
Sohail Somani