[serialization] My implementation of serialization of STL iterators.

I succeeded in writing code that successfully serializes my data structures, which contain a LOT a iterators, and even vectors of iterators. However, the end result turned out to be extremely slow (it takes several minutes), due to some inefficiency in boost::serialization (it should be possible to serialize this in a few seconds). I didn't investigate why boost::serialization becomes so slow, but I suppose it is caused by the fact that as a result of how things are serialized, the structure written to file is several thousands of levels deep (meaning, that in the XML file the indentation of the output is also many thousands of TAB's deep). I'm going to approach my problem of serialization in a different way now - making a copy of the data structure, such that the result can be serialized without needing to serialize iterators. Nevertheless, I don't want to keep my solution for the iterators from others; so below follows a brief overview of what I made. An average serialize() function looks as follows: template<class Archive> void Data::serialize(Archive& ar, unsigned int const) { ar & BOOST_SERIALIZATION_BASE_OBJECT_NVP(BaseClass); ar & BOOST_SERIALIZATION_NVP(M_variable); ar & BOOST_SERIALIZATION_ITERATOR_NVP(container, M_iterator); ar & BOOST_SERIALIZATION_ITERATOR_NVP(container, M_vector_of_iterators); } where BOOST_SERIALIZATION_ITERATOR_NVP is my invention ;). 'container' has to be the container that the iterator(s) point at. In the case of a vector of iterators, that is the container that the iterators point at, not the vector. Ie, M_iterator could of type std::set<FooBar>::iterator, then container is the std::set<FooBar> container, and M_vector_of_iterators would be of type std::vector<std::set<FooBar>::iterator>. Both, the container type (std::set<FooBar>) as well as the type of it's elements need to be tracked; hence, the above would need: BOOST_CLASS_TRACKING(FooBar, boost::serialization::track_always); BOOST_CLASS_TRACKING(std::set<FooBar>, boost::serialization::track_always); This would then first serialize the container (if not already done) and then serialize the element that the iterator points to (if not already done). I defined BOOST_SERIALIZATION_ITERATOR_NVP as follows: // This macro generates the name string (from 'name'). #define BOOST_SERIALIZATION_ITERATOR_NVP(container, name) \ serialization_iterator_nvp(container, BOOST_PP_STRINGIZE(name), name) where serialization_iterator_nvp is an heavily overload, inline template function: template<typename Container> inline boost::serialization::nvp<IteratorWrapper<Container, typename Container::iterator> > serialization_iterator_nvp(Container& container, char const* name, typename Container::iterator const& t) { IteratorWrapper<Container, typename Container::iterator> tmp(container, const_cast<typename Container::iterator&>(t)); return boost::serialization::nvp<IteratorWrapper<Container, typename Container::iterator> >(name, tmp); } template<typename Container> inline boost::serialization::nvp<IteratorWrapper<Container, typename Container::const_iterator> > serialization_iterator_nvp(Container& container, char const* name, typename Container::const_iterator const& t) { IteratorWrapper<Container, typename Container::const_iterator> tmp(container, const_cast<typename Container::const_iterator&>(t)); return boost::serialization::nvp<IteratorWrapper<Container, typename Container::const_iterator> >(name, tmp); } // Specialization for std::set, because for that container, the iterator and const_iterator are the same type! template<typename T> inline boost::serialization::nvp<IteratorWrapper<std::set<T>, typename std::set<T>::const_iterator> > serialization_iterator_nvp(std::set<T>& container, char const* name, typename std::set<T>::const_iterator const& t) { IteratorWrapper<std::set<T>, typename std::set<T>::const_iterator> tmp(container, const_cast<typename std::set<T>::const_iterator&>(t)); return boost::serialization::nvp<IteratorWrapper<std::set<T>, typename std::set<T>::const_iterator> >(name, tmp); } And in order for BOOST_SERIALIZATION_ITERATOR_NVP to also deal with vectors of iterators: template<typename Container> inline boost::serialization::nvp<VectorIteratorWrapper<Container, typename Container::iterator> > serialization_iterator_nvp(Container& container, char const* name, std::vector<typename Container::iterator> const& v) { VectorIteratorWrapper<Container, typename Container::iterator> tmp(container, const_cast<std::vector<typename Container::iterator>&>(v)); return boost::serialization::nvp<VectorIteratorWrapper<Container, typename Container::iterator> >(name, tmp); } template<typename Container> inline boost::serialization::nvp<VectorIteratorWrapper<Container, typename Container::const_iterator> > serialization_iterator_nvp(Container& container, char const* name, std::vector<typename Container::const_iterator> const& v) { VectorIteratorWrapper<Container, typename Container::const_iterator> tmp(container, const_cast<std::vector<typename Container::const_iterator>&>(v)); return boost::serialization::nvp<VectorIteratorWrapper<Container, typename Container::const_iterator> >(name, tmp); } // Specialization for std::set, because for that container, the iterator and const_iterator are the same type! template<typename T> inline boost::serialization::nvp<VectorIteratorWrapper<std::set<T>, typename std::set<T>::const_iterator> > serialization_iterator_nvp(std::set<T>& container, char const* name, std::vector<typename std::set<T>::const_iterator> const& v) { VectorIteratorWrapper<std::set<T>, typename std::set<T>::const_iterator> tmp(container, const_cast<std::vector<typename std::set<T>::const_iterator>&>(v)); return boost::serialization::nvp<VectorIteratorWrapper<std::set<T>, typename std::set<T>::const_iterator> >(name, tmp); } This uses two template class wrappers (which are declared before these functions in my file): IteratorWrapper and VectorIteratorWrapper. Their definitions are as follows: // A wrapper for iterators. template<class Container, typename Iterator> class IteratorWrapper { public: IteratorWrapper(Container& container, Iterator& iter) : M_container(container), M_iter(iter) { } private: Container& M_container; // The container. Iterator& M_iter; // The actual iterator. public: template<class Archive> void save(Archive& ar, unsigned int const) const { BOOST_STATIC_ASSERT(boost::serialization::tracking_level<Container>::type::value == boost::serialization::track_always); BOOST_STATIC_ASSERT(boost::serialization::tracking_level<typename Container::value_type>::type::value == boost::serialization::track_always); ar << boost::serialization::make_nvp("container", M_container); bool is_end = M_iter == M_container.end(); ar << boost::serialization::make_nvp("end", is_end); if (!is_end) ar << boost::serialization::make_nvp("element", *M_iter); } template<class Archive> void load(Archive& ar, unsigned int const) { ar >> boost::serialization::make_nvp("container", M_container); bool is_end; ar >> boost::serialization::make_nvp("end", is_end); if (!is_end) { typename Container::value_type* tmp_value; ar >> boost::serialization::make_nvp("element", tmp_value); } else M_iter = M_container.end(); } BOOST_SERIALIZATION_SPLIT_MEMBER() }; // A wrapper for a vector of iterators. template<class Container, typename Iterator> class VectorIteratorWrapper { public: VectorIteratorWrapper(Container& container, std::vector<Iterator>& iter) : M_container(container), M_vec(iter) { } private: Container& M_container; // The container. std::vector<Iterator>& M_vec; // The actual vector of iterators. public: template<class Archive> void save(Archive& ar, unsigned int const) const { BOOST_STATIC_ASSERT(boost::serialization::tracking_level<Container>::type::value == boost::serialization::track_always); BOOST_STATIC_ASSERT(boost::serialization::tracking_level<typename Container::value_type>::type::value == boost::serialization::track_always); int count = M_vec.size(); ar << BOOST_SERIALIZATION_NVP(count); for (typename std::vector<Iterator>::const_iterator iter = M_vec.begin(); iter != M_vec.end(); ++iter) { IteratorWrapper<Container, Iterator> iter_functor(M_container, const_cast<Iterator&>(*iter)); ar << boost::serialization::make_nvp("item", iter_functor); } } template<class Archive> void load(Archive& ar, unsigned int const) { int count; ar >> BOOST_SERIALIZATION_NVP(count); while(count--) { Iterator tmp_iter; IteratorWrapper<Container, Iterator> iter_functor(M_container, tmp_iter); ar >> boost::serialization::make_nvp("item", iter_functor); M_vec.push_back(tmp_iter); } } BOOST_SERIALIZATION_SPLIT_MEMBER() }; In both cases, the assertions make sure you didn't forget the two BOOST_CLASS_TRACKING lines. Have fun with it, Carlo Wood <carlo@alinoe.com> PS I didn't test the load() functions, but the resulting archive (I used xml) looked correct, thus, the serialization works.
participants (1)
-
Carlo Wood