Re: [Boost-users] Serialization: BOOST_CLASS_EXPORT changes between1.38 and 1.52

On 02/07/13 10:36, Robert Ramey wrote:

Nathan Whitehorn wrote:

[trimmed]

...

So my questions: 1. Is it possible for things that would return GUIDs of NULL to try harder and look in a global registry instead of silently breaking things? This kind of global lookup was how 1.38 always worked and it seems considerably less fragile.

The old method did instantiation by default so it worked then in some cases where the current one won't. So it seems "less fragile". But I think that's sort of an illusion. It does so at a cost of gratuitous instantiations which often are harmless - though non-optimal. But the real problem is that it left this out of the hands of the programmer. This could lead to silent and surprising behavior. Now we have the situtation where this behavior can't happen - we have to explicitly plan for it. I believe that this leads to less surprising programs - albiet at the cost of some surprising behavior at build time.

Thanks for the explanation! What I'm running into is surprising behavior at *run time* despite a build that apparently works. We've managed to avoid all the compile and link time redefinition issues so far by some combination of planning and luck.

...

2. Is there a way to handle BOOST_CLASS_EXPORT_KEY() sanely in the case of templates without the risk of silent serialization failures -- in all instances of that class -- that depend on global initialization order?

I believe that the best way to do this is to just do an explicit instantiation in a cpp file which imports the header containing BOOST_CLASS_EXPORT_KEY() and includes BOOST_CLASS_EXPORT_IMPL(). Once compiled, this can be added to a library or DLL. This will result in one and only instance of the class serialization existing in the program rather than mulitple ones (in the case of DLLS). Less code and better yet, this eliminates the possibility that the mainline module and the dll have different versions of the code which would be agony of the worst type to track down.

This is basically what we were already doing (there were never BOOST_CLASS_EXPORT() or -- with the exception of some templated things -- inlineable serialize() routines in header files for the reasons you mention). What I'm concerned about is a situation like this: Main library: Header: template<typename T> class I3Vector : public std::vector<T>, public I3FrameObject (our base class) { private: serialize(); }; BOOST_CLASS_EXPORT_KEY(I3Vector<T>) for a variety of T Implementation: Second library (a reasonably standard part of the software): Header: BOOST_CLASS_EXPORT_KEY(I3Vector<A>) for some other type A Implementation: Third library (written by someone else as an addon): Implementation: Serialize an I3Vector<A> *without* including the second library's header What happens here is that library #3 will usually appear to work -- and certainly compile and link without issue -- because all the serialization instantiation was done in library #2. *However*, if library 3 is loaded before library 2, both library 3 *and* 2's attempt to serialize I3Vector<A> will starting failing with an unregistered class exception. This is because extended_type_info_typeid<A> is a singleton and the first instance of it (as well as the now competing definitions of the classes if not fully inlined) came from library 3 where the GUID template specialization had not happened. This is an awful problem to have to debug, especially when libraries 2 and 3 are written by unrelated parties and the behavior happens to depend on the user's choice of load order (the libraries come in through RTLD at runtime). It means that all possible I3Vector<T> need to have keys exported in a common header somewhere that you can't possibly avoid including if you ever use an I3Vector<T>. Otherwise, everything can break everywhere if you happen not to have included that header in wherever happens to be the first occurrence of the type from the perspective of the (potentially dynamic) linker.

...

3. Is it possible to change the GUID set in the extended type info object of a pointer_[i/o]serializer at runtime after the class has been added to the export registry?

I have never considered this. I don't see what this would be used for. The singleton class table is never modified after it is constructed (before main is called). This is necessary for the serialization library to be thread-safe.

What I was hoping to do is to replace (still before main is called) any possible NULL GUIDs for a class with a non-NULL one if the relevant extended_type_info ever gets instantiated with a non-NULL GUID (which would require some changes to how the extended_type_info_typeid constructor works, but that's a separate issue). The idea would be that the GUID attached to a BOOST_CLASS_EXPORT(), when it runs, would be the final word on the matter instead of a leftover NULL from an instantiation in some place that didn't know about the GUID.

...

4. Are there any suggested mechanisms for local hacks, given that we control the archive implementation, to implement 1-3 without changes to boost?

to re-summarize my suggestion above.

a) change all the headers to use BOOST_CLASS_EXPORT_KEY() b) make a small *.cpp file for each header which imports the header and invokes BOOST_CLASS_EXPORT_IMPL(). c) add your small *.cpp file to your library - either static library or dll. d) while you're at it, you might want to consider adding the serialize, save, and load functions for the class to the *.cpp file and not making them inline. This will eliminate any code bloat generated by the serialization library. If your DLLS are dynamically loadable, they will only occupy memory when the the classes they refer to are actually being used at runtime. (just don't load/unload the DLLS while multi-threading - use a mutex!)

it seems you've touched upon the issue regarding serialization of template classes. This was also touched upon in a previous email. Currently we have to explicitly instantiate any templates we want to serialize. Automatically instantion of template generated classes using some combination of enable_if, partial specialization and who knows what else is interesting to consider, but likely much trickier than first meets eye. Also our "guid" is a string which can only be processed at runtime. Replacing this with a "guid" generated at compile time from the class name, might make somethings possible which weren't before. This is sort of irrelevant to your current situation, but I like to keep the pot boiling.

Thanks for the suggestions. They mostly reflect what we were already doing -- people get beaten with a stick if they try to instantiate serialize methods in header files or multiple times and they are usually kept in implementation files for that reason. We've mostly given up on automatic instantiation, but kept it easy to extend if you want to (see the problem with I3Vector above) -- although it would be amazing if you figured out how to do it. -Nathan