[Serialization] Inconsistent tracking_level when switching from static to DLL
Hi, I am using boost (1.40) serialization on Windows (VS 2008) to serialize a complex hierarchy of classes. My application is composed of different modules, each having its own serialization library. I have a Core library, with a CoreSerialization library, that defines a bunch of classes and base data types; I also have a Vehicle library, with a separate VehicleSerialization library, the defines some other classes with base classes in Core and that also use some user data types defined in Core. Using static libraries, everything saves and loads perfectly. We recently switched to using DLLs. When loading and saving with an application that only uses core (as Dlls) everything serializes perfectly. We can load files that were generated by an older application (using static libraries), save them and load them back. If the application also uses the Vehicle libraries, it can load older files (generated using static libraries), save them, but fails to load them back. The difference between the files generated with a DLL application and a static link application, lies in the tracking of certain classes. Since the serialization is split between 2 DLLs, the tracking level (by default selectively) will actually depends on what classes are serialized in the DLL and may be different in different DLLs. This seems to be cause by the various singleton that are created, notably pointer_oserializer<T> and oserializer<T>. In a statically linked application, there is only one instance on those singleton, in a DLL linked application, there is one instance of the singleton for each DLL. I've also noticed that in my Core DLL, for certain type, there are both notably pointer_oserializer<T> and oserializer<T>, while in the Vehicle DLL, there is only notably oserializer<T>, hence the difference in tracking level. There is an easy fix here, not to use selective tracking but track always or never, explicitly. But still, there seems to be some data that needs to be shared across various serialization DLLs. I am a bit worry about some other bugs popping out relative to it. Has this bug ever been found? It may be a side effect of putting serialization code in DLLs, the documentation on that subject is a bit sparse. Thanks for any inputs Guy -- Guy Prémont, D.Sc. Architecte logiciel senior / Senior software architect CM Labs Simulations Inc. blocked::http://www.cm-labs.com/ http://www.cm-labs.com/ Tel. 514-287-1166 ext. 237
Guy Prémont wrote:
Hi,
I am using boost (1.40) serialization on Windows (VS 2008) to serialize a complex hierarchy of classes.
My application is composed of different modules, each having its own serialization library. I have a Core library, with a CoreSerialization library, that defines a bunch of classes and base data types; I also have a Vehicle library, with a separate VehicleSerialization library, the defines some other classes with base classes in Core and that also use some user data types defined in Core. Using static libraries, everything saves and loads perfectly.
We recently switched to using DLLs. When loading and saving with an application that only uses core (as Dlls) everything serializes perfectly. We can load files that were generated by an older application (using static libraries), save them and load them back.
If the application also uses the Vehicle libraries, it can load older files (generated using static libraries), save them, but fails to load them back.
The difference between the files generated with a DLL application and a static link application, lies in the tracking of certain classes. Since the serialization is split between 2 DLLs, the tracking level (by default selectively) will actually depends on what classes are serialized in the DLL and may be different in different DLLs.
This seems to be cause by the various singleton that are created, notably pointer_oserializer<T> and oserializer<T>. In a statically linked application, there is only one instance on those singleton, in a DLL linked application, there is one instance of the singleton for each DLL. I've also noticed that in my Core DLL, for certain type, there are both notably pointer_oserializer<T> and oserializer<T>, while in the Vehicle DLL, there is only notably oserializer<T>, hence the difference in tracking level.
There is an easy fix here, not to use selective tracking but track always or never, explicitly. But still, there seems to be some data that needs to be shared across various serialization DLLs. I am a bit worry about some other bugs popping out relative to it.
Has this bug ever been found? It may be a side effect of putting serialization code in DLLs, the documentation on that subject is a bit sparse.
Thanks for any inputs
This is a result of the serialization library maybe being a little too smart for it's own good. The default behavior is to track if serialized as a pointer. This could vary if one invocation uses serialization of pointers and other doesn't. At one point I included code which would trap when the serialization code for same type appeard in more than one DLL. This fixed the problem, but it required that users organize their code so that serialization of a specific type be included in one and only one DLL. This turned out to be too onerous a requirement so I had to comment out this check. So, I'm going to guess that the safest thing is for such types is to either turn tracking off or on rather than using the default. Robert Ramey
This is a result of the serialization library maybe being a little too smart for it's own good.
The default behavior is to track if serialized as a pointer. This could vary if one invocation uses serialization of pointers and other doesn't. At one point I included code which would trap when the serialization code for same type appeard in more than one DLL. This fixed the problem, but it required that users organize their code so that serialization of a specific type be included in one and only one DLL. This turned out to be too onerous a requirement so I had to comment out this check.
So, I'm going to guess that the safest thing is for such types is to either turn tracking off or on rather than using the default.
Robert Ramey
Thanks, it is what I thought. I am curious about the trap to check if serialization code is in several DLLs. What kind of code organisation are you talking about? In my case, the serialization code for a class T is in only one DLL, but it is used called by each DLL that contains serialization for a class that uses class T. The method T::serialize is in only one DLL. However, if class A has a member of type T it would have template<typename Archive> void A::serialize(Archive& ar, int version) { ar & t; // where t is a T } As far as i see, it is the invocation of serialization in a serialize method (ar & t) that triggers the creation of the o/iserializer for type T. Is that it? In the case where t is a T*, the creation of a pointer_o/iserializer would be triggered. Thanks a lot for your input. Guy
Guy Prémont wrote:
This is a result of the serialization library maybe being a little too smart for it's own good.
The default behavior is to track if serialized as a pointer. This could vary if one invocation uses serialization of pointers and other doesn't. At one point I included code which would trap when the serialization code for same type appeard in more than one DLL. This fixed the problem, but it required that users organize their code so that serialization of a specific type be included in one and only one DLL. This turned out to be too onerous a requirement so I had to comment out this check.
So, I'm going to guess that the safest thing is for such types is to either turn tracking off or on rather than using the default.
Robert Ramey
Thanks, it is what I thought. I am curious about the trap to check if serialization code is in several DLLs. What kind of code organisation are you talking about?
In my case, the serialization code for a class T is in only one DLL, but it is used called by each DLL that contains serialization for a class that uses class T. The method T::serialize is in only one DLL.
However, if class A has a member of type T it would have template<typename Archive> void A::serialize(Archive& ar, int version) { ar & t; // where t is a T } As far as i see, it is the invocation of serialization in a serialize method (ar & t) that triggers the creation of the o/iserializer for type T. Is that it?
It is so you'd better use template<typename Archive> void A::serialize(Archive& ar, int version); // note ; and include the difiniiton in the DLL This is a murky area which I spent a fair amount of time investigating some time ago. Now I don't remember what my final conclusions were. I thought the problem manifested it self in as "unregistered class" exception so that in the absense of that, one could assume everyhing was kosher. Robert Ramey
Hi, I have work more on the problem of inconsistency of pointer tracking AND pointer serialization when using several DLLs that contains serialization code. Some pseudo-code is attached to illustrate the complete situation. In brief, a DLL_SERIALIZATION_CORE.dll contains archive types and the serialization code for several classes. A DLL_SERIALIZATION_EXTENSION.dll contains serialization code for additional classes. Classes serialized in the extension DLL use some object from the core DLL. When building and linking with static library, all is working fine. CORE.dll serializes class A as a pointer and as an object, both i/oserializer<> and pointer_oserializer<> singleton are created. EXTENSION.dll serializes class A as an object only, i/oserializer<> singleton is created. Depending on the ordering of serialization during loading, it happens that basic_iarchive_impl::register_type() overwrite the pointer_iserializer that was set in by the CORE dll. When saving, automatic tracking level is not correctly deduced since the pair oserializer<>/pointer_oserializer<> is created per DLL. There seem to be a clear need for a centralized repository of i/oserializer for each type, so that all DLLs use the same ones. In addition, using type_info::before() to order a std::map<> does not seem to be working. At least with windows VC9, before() just orders according to the address of typeinfo*; since a DLL creates a typeinfo object for each type it contains, using the same type in different DLLs can not be ordered with typeinfo::before() I've been working with Boost.Serialization for a while now. I'm really impressed with the ease of use, the scalability, and the power of the library. However, splitting the serialization code in several DLLs, although working for simple cases, have shown several problems. In static libs, boost serialization is extremely robust, in DLLs, several issues needs to be addressed to have the same robustness. Looking at my pseudo-code, maybe that I am not using the library as I should. I would be grateful for any hint or information. Thanks Guy -- Guy Prémont, D.Sc. Architecte logiciel senior / Senior software architect CM Labs Simulations Inc. http://www.cm-labs.com/ Tel. 514-287-1166 ext. 237
-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users- bounces@lists.boost.org] On Behalf Of Robert Ramey Sent: Wednesday, December 01, 2010 3:21 PM To: boost-users@lists.boost.org Subject: Re: [Boost-users] [Serialization] Inconsistenttracking_level whenswitching from static to DLL
Guy Primont wrote:
This is a result of the serialization library maybe being a little too smart for it's own good.
The default behavior is to track if serialized as a pointer. This could vary if one invocation uses serialization of pointers and other doesn't. At one point I included code which would trap when the serialization code for same type appeard in more than one DLL. This fixed the problem, but it required that users organize their code so that serialization of a specific type be included in one and only one DLL. This turned out to be too onerous a requirement so I had to comment out this check.
So, I'm going to guess that the safest thing is for such types is to either turn tracking off or on rather than using the default.
Robert Ramey
Thanks, it is what I thought. I am curious about the trap to check if serialization code is in several DLLs. What kind of code organisation are you talking about?
In my case, the serialization code for a class T is in only one DLL, but it is used called by each DLL that contains serialization for a class that uses class T. The method T::serialize is in only one DLL.
However, if class A has a member of type T it would have template<typename Archive> void A::serialize(Archive& ar, int version) { ar & t; // where t is a T } As far as i see, it is the invocation of serialization in a serialize method (ar & t) that triggers the creation of the o/iserializer for type T. Is that it?
It is so you'd better use
template<typename Archive> void A::serialize(Archive& ar, int version); // note ;
and include the difiniiton in the DLL
This is a murky area which I spent a fair amount of time investigating some time ago. Now I don't remember what my final conclusions were. I thought the problem manifested it self in as "unregistered class" exception so that in the absense of that, one could assume everyhing was kosher.
Robert Ramey
Guy Prémont wrote:
Hi,
I have work more on the problem of inconsistency of pointer tracking AND pointer serialization when using several DLLs that contains serialization code.
Some pseudo-code is attached to illustrate the complete situation. In brief, a DLL_SERIALIZATION_CORE.dll contains archive types and the serialization code for several classes. A DLL_SERIALIZATION_EXTENSION.dll contains serialization code for additional classes. Classes serialized in the extension DLL use some object from the core DLL.
When building and linking with static library, all is working fine.
CORE.dll serializes class A as a pointer and as an object, both i/oserializer<> and pointer_oserializer<> singleton are created. EXTENSION.dll serializes class A as an object only, i/oserializer<> singleton is created.
Depending on the ordering of serialization during loading, it happens that basic_iarchive_impl::register_type() overwrite the pointer_iserializer that was set in by the CORE dll.
When saving, automatic tracking level is not correctly deduced since the pair oserializer<>/pointer_oserializer<> is created per DLL.
There seem to be a clear need for a centralized repository of i/oserializer for each type, so that all DLLs use the same ones. In addition, using type_info::before() to order a std::map<> does not seem to be working. At least with windows VC9, before() just orders according to the address of typeinfo*; since a DLL creates a typeinfo object for each type it contains, using the same type in different DLLs can not be ordered with typeinfo::before()
I've been working with Boost.Serialization for a while now. I'm really impressed with the ease of use, the scalability, and the power of the library. However, splitting the serialization code in several DLLs, although working for simple cases, have shown several problems. In static libs, boost serialization is extremely robust, in DLLs, several issues needs to be addressed to have the same robustness.
Looking at my pseudo-code, maybe that I am not using the library as I should. I would be grateful for any hint or information.
Thanks Guy
As I've said before, I believe that this occurs because the code to serialize some data types is included in more than one code module (dll or exe). I would re-organize my code so that this does not occur. There is code in the library which will detect when code to serialize particular data type is included in more than one execution module (dll). This code will trap with an exception when a dll with duplicate code is loaded. This would enforce and detect violations of the above rule. Unfortunately, I had to comment out this code because many found this requirement to hard to implement and didn't have any problems. I would recommend finding and uncommenting this code and rebuilding the library. This should trap violations of the above and help you re-organize the code to be rock-solid. If I had nothing else to do, I would uncomment the code to re-enable the trap and add a runtime archive switch (like no_header) to suppress the trap for those who wish to. But, I have other stuff to do right now. Of course, it could be an entirely different problem, but as long as the current situation exists, I can only speculate. Robert Ramey
participants (2)
-
Guy Prémont -
Robert Ramey