serialization support for std::complex and boost::ublas::vector
Here are a couple of useful additions for serialization. std::complex is pretty trivial, and a no-brainer IMHO. I implemented support for ublas::vector unbounded_array only. Would be nice to do more, but that's all I need at the moment. #include <boost/archive/binary_oarchive.hpp> #include <boost/archive/binary_iarchive.hpp> #include <boost/serialization/split_free.hpp> template<typename Archive, typename T> inline void save (Archive &ar, const std::complex<T>& z, const unsigned int) { ar << real (z); ar << imag (z); } template<typename Archive, typename T> inline void load (Archive &ar, std::complex<T>& z, const unsigned int) { T i, r; ar >> r; ar >> i; z = std::complex<T> (r, i); } namespace boost { namespace serialization { template<class Archive, class T> inline void serialize (Archive &ar, std::complex<T>& z, const unsigned int file_version) { boost::serialization::split_free (ar, z, file_version); } } } template<class Archive, class U> inline void save (Archive &ar, const vector<U> &v, const unsigned int) { unsigned int count = v.size(); ar << count; typename vector<U>::const_iterator it = v.begin(); while (count-- > 0) { ar << *it++; } } template<class Archive, class U> inline void load (Archive &ar, vector<U> &v, const unsigned int) { unsigned int count; ar >> count; v.resize (count); typename vector<U>::iterator it = v.begin(); while (count-- > 0) { ar >> *it++; } } namespace boost { namespace serialization { template<class Archive, class U> inline void serialize (Archive &ar, vector<U>& v, const unsigned int file_version) { boost::serialization::split_free (ar, v, file_version); } } }
A couple of questions/observations: a) ...
I presume that real imag return either primitives(float or double) or references to these. So that tracking would not be an issue.
Hmm, loading to a temporary and then moving to a final destination could create an issue with tracking. There are a couple of possibilities: * if real/imag return references, then reformulate the above to: ar >> real(z); ar >> imag(z); since this is now symetric with the save one could avoid the split entirely and simplify it even more. * use reset_object_address b) what does "vector" refer to. it has no namespace associated with it. Could this not easily be confused with std::vector?
... I would like to see the following: a) a better/more complete test for std::complex. This would be easy - just use the current tests as a template. The std::complex serialization could be placed into boost/serialization/utility?/complex.hpp - ? whatever std uses. b) vector I presume is uBlas vector and would be added to the uBlas namespace/directory tree. Robert Ramey
Robert Ramey wrote:
You and I both wish real() and imag() returned a ref!!! But they don't. So I have: template<typename Archive, typename T> inline void load (Archive &ar, std::complex<T>& z, const unsigned int) { T i, r; ar >> boost::serialization::make_nvp ("real", r); ar >> boost::serialization::make_nvp ("imag", i); z = std::complex<T> (r, i); } Now how can I use reset_object_address, now that the the addresses of the real and imag pieces of the constructed object are no longer accessable?
Neal Becker wrote:
Looking at this more carefully, and realizine that real and imag return values rather than references, I think I gave the wrong answer. inline void load (Archive &ar, std::complex<T>& z, const unsigned int) { T i, r; ar >> boost::serialization::make_nvp ("real", r); ar >> boost::serialization::make_nvp ("imag", i); z = std::complex<T> (r, i); } I think the above will work just fine. The address being tracked is that of z so I expect that this will work exactly as one would expect. The saving would be exactly as you had it before. The only remaining question is what serialization traits - if any - should be assigned to std::complex<T>. Tracked/non-tracked. implementation level - primitive, object, with class-id, etc. My guess would be non-tracked primitive as that would more closely fit with other types such as float etc. I see no reason why any class version needs to be stored either. Robert Ramey
Robert Ramey wrote:
Thanks. I believe I have this OK now, but it was _much_ harder than it should have been, because the only documentation is wrong. I tried to follow the example here: http://boost.org/libs/serialization/doc/traits.html#templates But it looks like the correct implementation is this: namespace boost { namespace serialization { template<class T> struct implementation_level<std::complex<T> > { // typedef mpl::int_<object_serializable> type; typedef mpl::integral_c_tag tag; typedef mpl::int_<primitive_type> type; BOOST_STATIC_CONSTANT( int, value = implementation_level::type::value ); }; // // nvp objects are generally created on the stack and are never tracked template<class T> struct tracking_level<std::complex<T> > { typedef mpl::integral_c_tag tag; typedef mpl::int_<track_never> type; BOOST_STATIC_CONSTANT( int, value = tracking_level::type::value ); BOOST_STATIC_ASSERT(( mpl::greater< /* that is a prmitive */ implementation_level< T >, mpl::int_<primitive_type> >::value )); }; } } This is what I found in level.hpp and tracking.hpp
I checked the documentation and I see your point. I'll make adjustments to it. Considering more carefully what traits should be std::complex<T> I'm thinking now they should be: implementation level - object_serializable. This will guarentee that your serialization functions are called rather than using stream i/o but will not add class id or versioning information for the class. This presumes that the method of serialization for this std::complex will not change in the future - a fair assumption I believe. tracking - track_never I see std::complex as a type used in disparte places in the application and akin to float/complex etc. So I would mark this as track_never. Robert Ramey
Robert Ramey wrote:
OK, so how does this look? This implements std::complex and ublas::vector. Does not yet support allocator paramater to ublas::vector, but I don't think that would be difficult to add. namespace ublas = boost::numeric::ublas; namespace boost { namespace serialization { template<class T> struct implementation_level<std::complex<T> > { typedef mpl::integral_c_tag tag; typedef mpl::int_<object_serializable> type; BOOST_STATIC_CONSTANT( int, value = implementation_level::type::value ); }; template<class T> struct tracking_level<std::complex<T> > { typedef mpl::integral_c_tag tag; typedef mpl::int_<track_never> type; BOOST_STATIC_CONSTANT( int, value = tracking_level::type::value ); }; } } template<typename Archive, typename T> inline void save (Archive &ar, const std::complex<T>& z, const unsigned int) { ar << boost::serialization::make_nvp ("real", const_cast<const T&>(real (z))); ar << boost::serialization::make_nvp ("imag", const_cast<const T&>(imag (z))); } template<typename Archive, typename T> inline void load (Archive &ar, std::complex<T>& z, const unsigned int) { T i, r; ar >> boost::serialization::make_nvp ("real", r); ar >> boost::serialization::make_nvp ("imag", i); z = std::complex<T> (r, i); } namespace boost { namespace serialization { template<class Archive, class T> inline void serialize (Archive &ar, std::complex<T>& z, const unsigned int file_version) { boost::serialization::split_free (ar, z, file_version); } } } template<class Archive, class U> inline void save (Archive &ar, const ublas::vector<U> &v, const unsigned int) { unsigned int count = v.size(); ar << count; typename ublas::vector<U>::const_iterator it = v.begin(); while (count-- > 0) { ar << *it++; } } template<class Archive, class U> inline void load (Archive &ar, ublas::vector<U> &v, const unsigned int) { unsigned int count; ar >> count; v.resize (count); typename ublas::vector<U>::iterator it = v.begin(); while (count-- > 0) { ar >> *it++; } } namespace boost { namespace serialization { template<class Archive, class U> inline void serialize (Archive &ar, ublas::vector<U>& v, const unsigned int file_version) { boost::serialization::split_free (ar, v, file_version); } } }
Neal Becker wrote:
Robert Ramey wrote:
...
This doesn't seem right to me. real(z) returns a value. I don't see how this can be cast to reference to a const value without a lot of (conforming) compilers objecting. I believe the correct way will be: const T r = real(z); ar << boost::serialization::make_nvp ("real", r); const T i = imag(z); ar << boost::serialization::make_nvp ("imag", i);
I wonder about this - but I know nothing of ublas. If v.resize actually creates the entries I guess it would be just fine. Now I wonder about my own implementation of serialization for std::vector. It seems yours here is more efficient. Robert Ramey
Robert Ramey wrote:
You're right! I forgot, real() and imag() have been changed to now return a ref (or const ref). This works with gcc-4.0.1, at least. Does this look OK? Does the ublas::vector stuff also need some nvp added? namespace boost { namespace serialization { template<class T> struct implementation_level<std::complex<T> > { typedef mpl::integral_c_tag tag; // typedef mpl::int_<primitive_type> type; typedef mpl::int_<object_serializable> type; BOOST_STATIC_CONSTANT( int, value = implementation_level::type::value ); }; // // nvp objects are generally created on the stack and are never tracked template<class T> struct tracking_level<std::complex<T> > { typedef mpl::integral_c_tag tag; typedef mpl::int_<track_never> type; BOOST_STATIC_CONSTANT( int, value = tracking_level::type::value ); }; } } template<typename Archive, typename T> inline void save (Archive &ar, const std::complex<T>& z, const unsigned int) { ar << boost::serialization::make_nvp ("real", real (z)); ar << boost::serialization::make_nvp ("imag", imag (z)); } template<typename Archive, typename T> inline void load (Archive &ar, std::complex<T>& z, const unsigned int) { ar >> boost::serialization::make_nvp ("real", real(z)); ar >> boost::serialization::make_nvp ("imag", imag(z)); } namespace boost { namespace serialization { template<class Archive, class T> inline void serialize (Archive &ar, std::complex<T>& z, const unsigned int file_version) { boost::serialization::split_free (ar, z, file_version); } } } template<class Archive, class U> inline void save (Archive &ar, const ublas::vector<U> &v, const unsigned int) { unsigned int count = v.size(); ar << count; typename ublas::vector<U>::const_iterator it = v.begin(); while (count-- > 0) { ar << *it++; } } template<class Archive, class U> inline void load (Archive &ar, ublas::vector<U> &v, const unsigned int) { unsigned int count; ar >> count; v.resize (count); typename ublas::vector<U>::iterator it = v.begin(); while (count-- > 0) { ar >> *it++; } } namespace boost { namespace serialization { template<class Archive, class U> inline void serialize (Archive &ar, ublas::vector<U>& v, const unsigned int file_version) { boost::serialization::split_free (ar, v, file_version); } } }
Neal Becker wrote:
well if imag and real return const an non-const references to T then you can simplify things to: namespace boost { namespace serialization { template<class Archive, class T> inline void serialize (Archive &ar, std::complex<T>& z, const unsigned int file_version) { ar & real(z); ar & imag(z); } } } Robert Ramey
Robert Ramey wrote:
OK, How about this then? namespace boost { namespace serialization { template<class T> struct implementation_level<std::complex<T> > { typedef mpl::integral_c_tag tag; // typedef mpl::int_<primitive_type> type; typedef mpl::int_<object_serializable> type; BOOST_STATIC_CONSTANT( int, value = implementation_level::type::value ); }; template<class T> struct tracking_level<std::complex<T> > { typedef mpl::integral_c_tag tag; typedef mpl::int_<track_never> type; BOOST_STATIC_CONSTANT( int, value = tracking_level::type::value ); }; } } namespace boost { namespace serialization { template<class Archive, class T> inline void serialize (Archive &ar, std::complex<T>& z, const unsigned int file_version) { ar & boost::serialization::make_nvp ("real", real(z)); ar & boost::serialization::make_nvp ("imag", imag(z)); // ar & real(z); // ar & imag(z); } } }
Hi, I just ran into a nasty thing with Boost.Serialization. While restructuring some of our base libraries to get rid of unnecessary headers and reduce header dependencies in general, I suddenly got a hard-to-track-down #error in basic_archive.hpp saying that I need to include archive headers before other serialization headers. I remember a discussion on this list about this subject in the past. I quickly skimmed through the thread and can only make the same plea as others have made in that thread: Please, please, please get rid of this header order dependency. In larger projects such header order dependencies are incredibly hard to get right. I also don't see why I shouldn't be able to do the following: // A.hpp: #include <boost/serialization/access.hpp> #include <boost/serialization/nvp.hpp> class A { public: A(float value) : data(value) {} private: friend class boost::serialization::access; template <typename ArchiveT> void serialize(ArchiveT& archive, const unsigned int version) { archive << BOOST_SERIALIZATION_NVP(data); } private: float data; }; // UsingA.cpp: #include "A.hpp" // include first to make sure it includes whatever it needs #include <boost/archive/polymorphic_binary_oarchive.hpp> ... This technique is often used to make sure that headers can be translated "on their own", i.e. that they include everything they need. Another example that demonstrates how hard it can be to get the header order right: // B.hpp: #include <A.hpp> #include <boost/archive/polymorphic_oarchive.hpp> // ^-- BOOM! A.hpp includes non-archive serialization headers class B { public: void Save(boost::archive::polymorphic_oarchive& archive) const; private: A a; }; Of course I can swap the two includes, but why should I care that A.hpp includes serialization headers. Also note that this is a contrived example. It could be that not A.hpp is the culprit, but any other header included by A.hpp might include serialization headers. This makes it very hard to find the correct header order. Also, assuming A.hpp comes from another library that I have not written. Why should I know about its implementation details. Why should I have to care in what order I need to include headers from another library. So again, please, whatever the cause for introducing this header order dependency is, get rid of it. It wasn't there in Boost.Serialization 1.32.0 and it shouldn't be in 1.33.0 either. Best Regards, Martin TAB Austria Haiderstraße 40 4052 Ansfelden Austria Phone: +43 7229 78040-218 Fax: +43 7229 78040-209 E-mail: martin.ecker@tab.at http://www.tab.at
This has been discussed on another thread. I've looked at this again and I believe this can be fixed. Its a bit more involved than meets the eye however. I haven't found an easy fix that addresses the competing requirements but I am hopeful that a solution can be found. Robert Ramey martin.ecker@tab.at wrote:
On Aug 29, 2005, at 10:00 AM, Robert Ramey wrote:
Is autolinking the only reason for these dependencies? If so, let's yank out autolinking of Serialization until we know we can fix the problem without include order dependencies. We've been talking about releasing a 1.33.1 (beta) to address the problems in 1.33.0, and this issue has quickly elevated itself to top priority. Doug
I hope not: auto-linking doesn't introduce any header dependencies of it's own. I'm paraphrasing but I believe there are two issues here: 1) Some serialisation headers have to be included in a certain order, that's a bad thing IMO, and yanking out auto-linking support won't address that. 2) The headers that include the auto-linking code, can be included by clients that don't actually need to link to the library. There's no clear cut serialisation header that's only included when you need to link to the extern lib. I believe that both issues should be solvable with header refactoring: as far as auto-linking goes, it means that the declarations for objects that are placed in the compiled lib need to be factored out into separate headers. Robert: is there any help you need from me with the auto-linking code? Anything that you don't understand or is causing problems here? HTH, John.
John Maddock wrote:
This is basically the problem. header extended_type_info can be included in places where not all the functions are in fact called. So normally there would be no need to link the library. OTOH, just including the header provokes the auto-linking. I'm currently looking at how to factor this into two pieces - one that is header only and one that requires linking to a library. This is made somewhat more tricky that first meets the eye due to the fact the the serialization library is pretty large and that extended_type_info in used in several places for similar but unrelated purposes.
Robert: is there any help you need from me with the auto-linking code? Anything that you don't understand or is causing problems here?
In the course of implementing auto-linking in the serialization library I became quite familiar with the auto-linking facility - much more than I thought I would have to be. In large part this is due to the design of the serialization libary. Its actually three pieces a) basic faciliies - e.g. extended type info, void_cast, etc b) basic archive facilities - serialization.lib (which includes a) above) b) archive faciltiy for wide char i/o - wserialization.lib. This latter imports references from b) above since that library/DLL contains the archive base class implementations. So it was quite an effort to keep these things from stepping all over each other. The final result is quite clean and I only have this problem left which I am working on now. Robert Ramey
I don't think anything that drastic will be necessary or desirable. The problem came about as a side effect of implementing auto-linking. As I've said, I believe the issue can be addressed to everyone's satisfaction, it just takes a little bit of patience. Robert Ramey Doug Gregor wrote:
Oh no - what you have is just perfect - assuming that resize creates new entries and doesn't just increase some internal indicator. A copy would create the problem of loading to a temporary object that is then copied from which would invalidate its address. Neal Becker wrote:
On Fri, 26 Aug 2005 17:32:51 -0400, Neal Becker wrote
...catching up on thread... Somehow I completely missed these flags when implementing the date-time serialization capabilities. Seems like the date-time classes should be set to primitive_types. But if I did this now, I'm going to break existing archives -- right?
Subsequent to this I studied a little bit more. I suggested that the std::complex type be marked as object_serializable. This means that the serialization functions are called but class verision information is not stored in the archive. This might be faster/more efficient then the default, but creates a problem if the serialization of the item is changed in the future. Given the simple nature of std::complex, I don't anticipate that happening. primitive would mean that the serialization functions aren't called at all and the archive handles them within the archive code. I also suggested that std::complex be untracked. That's because I see this as a type widely used accross different application areas. So it seems to me that this is correct. \>
Well, the default settings are pretty good - this probably why no one has ever mentioned it before. The question is whether date-time should include the class version in the seialized files. Given the non-obvious (to me anyway) nature of the implementation, it seems to me that details of its serialization could be refined in the future. Recording the class verision in the archive (The default, so its being done now) seems correct to me. Tracking isn't so clear. Right now all date-time instances are being tracked if its used anywhere as a pointer. This is the default. Not really a bad choice. But someone might say - well I'm using date/time in different parts of the applications and I don't want the fact that I'm serializing data/time as pointer in one part of the application creating a performance bottle neck in another part. But this isn't an issue with date/time but rather an issue with the concept of serialization traits in general. So I think data/time is just fine with the defaults. Robert Ramey
On Sun, 28 Aug 2005 09:13:10 -0700, Robert Ramey wrote
I expect the implementation of date-time serialization to evolve (specifically b/c I'm considering reworking some internals and dropping support for old compilers). However, we thought carefully about how these types should be serailized and no matter the internal implementation I don't expect the serialized form will change. So I really think the versioning is uneeded. Anyway, you didn't answer my question -- I presume that prior archives will be broken if versioning is switched off?
I'd say it would be pretty unusual to dynamically allocate and serialize a pointer to a date/time value. Date/time values tend to get used as value objects that are a part of more complex objects.
So I think data/time is just fine with the defaults.
Yeah, it's probably fine and will remain as is for now -- assuming my understanding about breaking existing archives is right. Just trying to see if there should be future optimizations that should be applied... Thx, Jeff
Jeff Garland wrote:
Anyway, you didn't answer my question -- I presume that prior archives will be broken if versioning is switched off?
ah - yes, I forgot there was a question. I would have to check on this. I know that if the version isn't included, you can't add it later. The idea that someone might want to move in the other direction never occured to me so off hand Ican't say what would happen. I suspect this wouldn't work for the same reason. If the achive contains class information but the code doesn't look for it, things would probably be out of sync. Robert Ramey
participants (7)
-
Doug Gregor -
Jeff Garland -
John Maddock -
martin.ecker＠tab.at -
Neal Becker -
Robert Ramey -
Russell Hind