Boost mpi, serialization and shared_ptr
Hi, I have started to parallelize one of my applications using MPI and I am running into trouble with serializing classes that contain shared_ptr member to be send through MPI. Below is a minimal case that demonstrates my problem. #include <boost/serialization/serialization.hpp> #include <boost/shared_ptr.hpp> #include <boost/mpi.hpp> #include <boost/serialization/shared_ptr.hpp> #include <boost/serialization/export.hpp> class Base { private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) { } public: Base() { } virtual ~Base() { } }; class Derived: public Base { private: double b; boost::shared_ptr<Base> MemberPointer; friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & boost::serialization::base_object<Base>(*this); ar & MemberPointer; ar & b; } public: Derived() { } virtual ~Derived() { } }; //BOOST_CLASS_EXPORT(Base) BOOST_CLASS_EXPORT(Derived) int main(int argc, char *argv[]) { boost::mpi::environment env(argc, argv); boost::mpi::communicator world; boost::shared_ptr<Base> Pointer(new Derived()); boost::mpi::broadcast(world, Pointer, 0); } I want to broadcast the derived class that has a member that is a shared_ptr. When I compile this code with gcc-4.6, intel or clang all compilers complain, this is the output from gcc-4.6 In file included from Hacks/serialization_mpi.cpp:11:0: /usr/local/include/boost/serialization/shared_ptr.hpp: In function ‘void boost::serialization::load(Archive&, boost::shared_ptr<U>&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = Base]’: /usr/local/include/boost/serialization/split_free.hpp:58:9: instantiated from ‘static void boost::serialization::free_loader<Archive, T>::invoke(Archive&, T&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ /usr/local/include/boost/serialization/split_free.hpp:74:5: instantiated from ‘void boost::serialization::split_free(Archive&, T&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ /usr/local/include/boost/serialization/shared_ptr.hpp:171:5: instantiated from ‘void boost::serialization::serialize(Archive&, boost::shared_ptr<U>&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = Base]’ /usr/local/include/boost/serialization/serialization.hpp:128:9: instantiated from ‘void boost::serialization::serialize_adl(Archive&, T&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ /usr/local/include/boost/archive/detail/iserializer.hpp:188:5: instantiated from ‘void boost::archive::detail::iserializer<Archive, T>::load_object_data(boost::archive::detail::basic_iarchive&, void*, unsigned int) const [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ Hacks/serialization_mpi.cpp:64:3: instantiated from here /usr/local/include/boost/serialization/shared_ptr.hpp:133:9: error: ‘class boost::mpi::packed_skeleton_iarchive’ has no member named ‘append’ /usr/local/include/boost/serialization/shared_ptr.hpp:139:5: error: ‘class boost::mpi::packed_skeleton_iarchive’ has no member named ‘reset’ If I remove the shared_ptr member, it compiles and runs. If I remove BOOST_CLASS_EXPORT(Derived) it compiles, but when running it I get terminate called after throwing an instance of 'boost::archive::archive_exception' what(): unregistered class - derived class not registered or exported Is this a bug or am I doing something wrong? A google search shows an old discussion (2007) that concludes that this problem is fixed in boost 1.45. I see this happening with both 1.48 and 1.49. Any help is appreciated. Regards Max
Max Moorkamp wrote:
before we get too carried away, please double check the documentation. the macro BOOST_SERIALIZATION_EXPORT has been upgraded to the more precise versions BOOST_SERIALIZATION_EXPORT_IMPLEMENT and BOOST_SERIALIZATION_EXPORT_KEY Make these tweaks and see if this helps. Robert Ramey
Thanks for your answer, I tried what you suggested. Now I have Derived.h #ifndef DERIVED_H_ #define DERIVED_H_ #include "Base.h" #include <boost/serialization/serialization.hpp> #include <boost/shared_ptr.hpp> #include <boost/serialization/shared_ptr.hpp> #include <boost/serialization/export.hpp> class Derived: public Base { private: double b; boost::shared_ptr<Base> MemberPointer; friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & boost::serialization::base_object<Base>(*this); ar & MemberPointer; ar & b; } public: Derived(); virtual ~Derived(); }; BOOST_CLASS_EXPORT_KEY(Derived) #endif /* DERIVED_H_ */ and Derived.cpp #include "Derived.h" #include <boost/mpi.hpp> #include <boost/serialization/shared_ptr.hpp> #include <boost/serialization/export.hpp> Derived::Derived() { } Derived::~Derived() { } BOOST_CLASS_EXPORT_IMPLEMENT(Derived) but now I get In file included from Derived.h:14, from Derived.cpp:8: /usr/local/include/boost/serialization/shared_ptr.hpp: In function ‘void boost::serialization::load(Archive&, boost::shared_ptr<U>&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = Base]’: /usr/local/include/boost/serialization/split_free.hpp:58: instantiated from ‘static void boost::serialization::free_loader<Archive, T>::invoke(Archive&, T&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ /usr/local/include/boost/serialization/split_free.hpp:74: instantiated from ‘void boost::serialization::split_free(Archive&, T&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ /usr/local/include/boost/serialization/shared_ptr.hpp:171: instantiated from ‘void boost::serialization::serialize(Archive&, boost::shared_ptr<U>&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = Base]’ /usr/local/include/boost/serialization/serialization.hpp:128: instantiated from ‘void boost::serialization::serialize_adl(Archive&, T&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ /usr/local/include/boost/archive/detail/iserializer.hpp:188: instantiated from ‘void boost::archive::detail::iserializer<Archive, T>::load_object_data(boost::archive::detail::basic_iarchive&, void*, unsigned int) const [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ Derived.cpp:25: instantiated from here /usr/local/include/boost/serialization/shared_ptr.hpp:155: error: ‘class boost::mpi::packed_skeleton_iarchive’ has no member named ‘reset’ If I omit the #include <boost/mpi.hpp> from Derived.cpp everything compiles, but as expected I get a runtime error terminate called after throwing an instance of 'boost::archive::archive_exception' what(): unregistered class - derived class not registered or exported Any ideas? Regards Max ________________________________________ From: boost-bounces@lists.boost.org [boost-bounces@lists.boost.org] On Behalf Of Robert Ramey [ramey@rrsd.com] Sent: 02 June 2012 03:30 To: boost@lists.boost.org Subject: Re: [boost] Boost mpi, serialization and shared_ptr Max Moorkamp wrote:
before we get too carried away, please double check the documentation. the macro BOOST_SERIALIZATION_EXPORT has been upgraded to the more precise versions BOOST_SERIALIZATION_EXPORT_IMPLEMENT and BOOST_SERIALIZATION_EXPORT_KEY Make these tweaks and see if this helps. Robert Ramey _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Hi, I guess this got drowned in the discussion about the beta release and the review of the multi-precision library. If anybody could tell me what I am doing wrong, any help is greatly appreciated.
I tried what you suggested. Now I have Derived.h #ifndef DERIVED_H_ #define DERIVED_H_ #include "Base.h" #include <boost/serialization/serialization.hpp> #include <boost/shared_ptr.hpp> #include <boost/serialization/shared_ptr.hpp> #include <boost/serialization/export.hpp> class Derived: public Base { private: double b; boost::shared_ptr<Base> MemberPointer; friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & boost::serialization::base_object<Base>(*this); ar & MemberPointer; ar & b; } public: Derived(); virtual ~Derived(); }; BOOST_CLASS_EXPORT_KEY(Derived) #endif /* DERIVED_H_ */ and Derived.cpp #include "Derived.h" #include <boost/mpi.hpp> #include <boost/serialization/shared_ptr.hpp> #include <boost/serialization/export.hpp> Derived::Derived() { } Derived::~Derived() { } BOOST_CLASS_EXPORT_IMPLEMENT(Derived) but now I get In file included from Derived.h:14, from Derived.cpp:8: /usr/local/include/boost/serialization/shared_ptr.hpp: In function ‘void boost::serialization::load(Archive&, boost::shared_ptr<U>&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = Base]’: /usr/local/include/boost/serialization/split_free.hpp:58: instantiated from ‘static void boost::serialization::free_loader<Archive, T>::invoke(Archive&, T&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ /usr/local/include/boost/serialization/split_free.hpp:74: instantiated from ‘void boost::serialization::split_free(Archive&, T&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ /usr/local/include/boost/serialization/shared_ptr.hpp:171: instantiated from ‘void boost::serialization::serialize(Archive&, boost::shared_ptr<U>&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = Base]’ /usr/local/include/boost/serialization/serialization.hpp:128: instantiated from ‘void boost::serialization::serialize_adl(Archive&, T&, unsigned int) [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ /usr/local/include/boost/archive/detail/iserializer.hpp:188: instantiated from ‘void boost::archive::detail::iserializer<Archive, T>::load_object_data(boost::archive::detail::basic_iarchive&, void*, unsigned int) const [with Archive = boost::mpi::packed_skeleton_iarchive, T = boost::shared_ptr<Base>]’ Derived.cpp:25: instantiated from here /usr/local/include/boost/serialization/shared_ptr.hpp:155: error: ‘class boost::mpi::packed_skeleton_iarchive’ has no member named ‘reset’ If I omit the #include <boost/mpi.hpp> from Derived.cpp everything compiles, but as expected I get a runtime error terminate called after throwing an instance of 'boost::archive::archive_exception' what(): unregistered class - derived class not registered or exported Any ideas? Regards Max
On 11/06/12 15:27, Max Moorkamp wrote:
Just an additional observation that might help to localize the problem. If I split the serialize function like this: friend class boost::serialization::access; template<class Archive> void save(Archive & ar, const unsigned int version) const { ar & boost::serialization::base_object<Base>(*this); ar & Pointer; ar & b; } template<class Archive> void load(Archive & ar, const unsigned int version) { ar & boost::serialization::base_object<Base>(*this); ar & Pointer; ar & b; } BOOST_SERIALIZATION_SPLIT_MEMBER() the code compiles (but crashes, of course) as long as I comment out the loading of the pointer. Max
Max Moorkamp wrote:
Here is he key. The serialization library has some special code to handle boosts beloved shared_ptr. This special code is found in boost/archive/shared_ptr_helper.hpp. it is added in at the "last possible moment" via public inheritance. see "boost\archive\binary_iarchive.hpp". This adds a special function called "reset". It seems that MPI serialization doesn't add in this functionality - hence the compile error. So the fix would be tweak the MPI serialization implementation to address this. It's not clear to me what this would entail. Longer term - some day - would be to add the functionaliy of a "generic helper" which any serialization implementation could use to permit serialization of otherwise unserializable classes. Robert Ramey
On 12/06/12 18:05, Robert Ramey wrote:
I see, looking at boost/mpi/packed_oarchive.hpp and boost/mpi/packed_iarchive.hpp I find class BOOST_MPI_DECL packed_iarchive : public iprimitive , public archive::detail::common_iarchive<packed_iarchive> , public archive::detail::shared_ptr_helper { and class BOOST_MPI_DECL packed_oarchive : public oprimitive , public archive::detail::common_oarchive<packed_oarchive> , public archive::detail::shared_ptr_helper { so that seems to be in order. Interestingly this: #include <iostream> #include <boost/shared_ptr.hpp> #include <boost/mpi.hpp> #include <boost/serialization/serialization.hpp> #include <boost/serialization/shared_ptr.hpp> class MyClass { private: double a; friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & a; } public: void SetA(double value) { a = value; } double GetA() { return a; } MyClass() { } virtual ~MyClass() { } }; int main(int argc, char *argv[]) { boost::mpi::environment env(argc, argv); boost::mpi::communicator world; boost::shared_ptr<MyClass> ClassPointer(new MyClass()); if (world.rank() == 0) { ClassPointer->SetA(5.0); } boost::mpi::broadcast(world, ClassPointer, 0); std::cout << "Rank: " << world.rank() << " " << ClassPointer->GetA() << std::endl; } compiles and works as expected. So it appears that the problem is not the serialization of shared_ptr as such, but only if it is a member of a class. Regards Max
participants (3)
-
Max Moorkamp -
Moorkamp, Max (Dr.)
-
Robert Ramey