Problems with De/Serializing data structures with dll
Hi, I'm quite noob with boost and I wrote a little test program to test how boost handles serialization of data structures and deserializing the data in dll-plug-in. ( In this case only linked list of BaseImplementation objects, but if I got everything OK more complex structures is need to be serialized). It seems that I have(surprisingly) some problems with de/serializing list properly maybe cause of m_next pointers. I've read the documentation, but not have found help for this issue. Here's some relevant parts of my code ( I know It's a bit ugly, but hey! It's only for test purposes :) : Base.h: /* Abstract base class declaration of objects in linked list*/ #ifndef BASE_H #define BASE_H #include <string> #include <iostream> #include <boost/serialization/assume_abstract.hpp> class Base { public: Base(){} virtual ~Base(){} virtual void setID(const int a_id) =0; virtual const int getID() const =0; virtual void setValue( const double a_value ) =0; virtual const double getValue() const =0; virtual void setDescription( const std::string a_desc ) =0; virtual const std::string getDescription() const =0; virtual void setNext( Base* a_next ) =0; virtual Base* getNext() =0; //Operation - prints some information regarding the object virtual void print()const =0; private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) { } }; BOOST_SERIALIZATION_ASSUME_ABSTRACT(Base) #endif ###################################### BaseImplementation.h: //Implementation of Base-class (cells of linked list) #ifndef BASEIMPLEMENTATION_H #define BASEIMPLEMENTATION_H #include <boost/archive/text_oarchive.hpp> #include <boost/archive/text_iarchive.hpp> //#include <boost/serialization/export.hpp> #include <boost/serialization/base_object.hpp> #include <boost/serialization/utility.hpp> #include "Base.h" class BaseImplementation : public Base { public: //Constructor & Destructor BaseImplementation(); BaseImplementation( int a_id ); ~BaseImplementation(); //setter & getter virtual void setID( const int a_id ); virtual const int getID() const; virtual void setValue( const double a_value ); virtual const double getValue() const; virtual void setDescription( const std::string a_desc ); virtual const std::string getDescription() const; virtual void setNext( Base* a_next ); virtual Base* getNext(); //Operation - print virtual void print() const; private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & boost::serialization::base_object<Base>(*this); ar & m_id; ar & m_value; ar & m_desc; //THIS SEEMS TO BE THE PROBLEM!!! ar & m_next; } int m_id; double m_value; std::string m_desc; Base* m_next; }; //BOOST_CLASS_EXPORT_GUID(BaseImplementation, "base_impl") #endif ############################################### main program: // PluginTest.cpp : Defines the entry point for the console application. #include "stdafx.h" #include "PluginInterface.h" #include "BaseImplementation.h" #include <iostream> #include <fstream> #include <ctime> #include <cstdio> int _tmain(int argc, _TCHAR* argv[]) { std::remove("serialized.plop"); ...foo... ...foo.. ...foo.. //--Serialization-------------------------------------------------- std::ofstream ofs("serialized.plop"); boost::archive::text_oarchive oa(ofs); BaseImplementation* myObject = NULL; Base* first = NULL; BaseImplementation* prev = NULL; Base* tmp = NULL; //starting point of time consumption calc. start = clock(); //Let's create linked list of 114 instances of //BaseImplementation for dll to deserialize for(int i=0; i < 114; i++) { myObject = new BaseImplementation(); //sets the next cell of the list if(prev) { prev->setNext(myObject); } //first cell of linked list else { first = myObject; tmp =myObject; } //sets the data for object myObject->setID( i ); myObject->setValue( 0.5 +(double)i ); myObject->setDescription("This is object"); prev = myObject; } //The serialization itself oa.register_type(static_cast<BaseImplementation *>(tmp)); for(int i=0; i < 114; i++) { oa << tmp; tmp = tmp->getNext(); //oa << *tmp; } //oa << first; ofs.close(); //Deallocates created & serialized objects for(int i=0; i < 114; i++) { tmp = first->getNext(); delete first; first =tmp; } //stop time consumption calc. stop = clock(); //time consumption of serialization long serializationTime = ( stop - start ); //---Serializatio ends here----------------------------------------- //---Deserialization------------------------------------------------ //start the clock start = clock(); //Deserializes the linked list created above for dll-plugin thePlugin->load("serialized.plop"); //stop the clock stop = clock(); //time consumption of deserialization long deserializationTime = ( stop - start ); //Prinst some deserialized data thePlugin->print(50); std::cout <<"Serializations takes: " <<serializationTime <<" ms" <<std::endl; std::cout <<"Deserialization takes: " <<deserializationTime <<" ms" <<std::endl; ########################################################## Definition of thePlugin class: #include ".\cmyplugin.h" cMyPlugin::cMyPlugin() : cIHAPlugin() { m_name = "My plugin!"; m_description = "A plugin which is located in DLL!"; BaseImplementation* m_baseImpl = NULL; } cMyPlugin::~cMyPlugin() { } bool cMyPlugin::install(cIHAPluginCoreInterface* a_pluginCore) { std::cout << "installing plugin.. "; if( !a_pluginCore ) { std::cout << "failed!\n"; return false; } std::cout << "success.\n"; return true; } /*! \brief Gets the plugin name. \return The plugin name. */ const char* cMyPlugin::getName() { return m_name.c_str(); } /*! \brief Gets the plugin description. \return The plugin description. */ const char* cMyPlugin::getDescription() { return m_description.c_str(); } const int cMyPlugin::getNumber() { return m_baseImpl->getID(); } void cMyPlugin::load(std::string file) { //opens archive std::ifstream ifs(file.c_str()); boost::archive::text_iarchive ia(ifs); ia.register_type<BaseImplementation>(); //this doesn't handle pointers to next cell in list ia >> m_baseImpl; ifs.close(); } void cMyPlugin::print(int a_index) { m_first =m_baseImpl; Base* tmp =m_first; if( !m_first ) { std::cout << "ULTIMATE ERROR!!" << std::endl; return; } else { std::cout <<"--- This is value(double) of first cell: "; std::cout << m_first->getValue() <<" ---" <<std::endl; std::cout <<"--- This is value(double) of second cell: "; std::cout <<m_first->getNext()->getValue() <<" ---" <<std::endl; } for(int i=0; i < 114; i++ ) { if(i == a_index ) { tmp->print(); break; } tmp =tmp->getNext(); } } ################################################## Ok, this works still fine with 114 or less instances on the list, but when I try to do this with more than 114 instances it fails on line: oa << tmp; when it's called for first time. Error Message: "Unhandled exception at 0x1029dfdc (msvcr90d.dll) in SerializationTest.exe: 0xC00000FD: Stack overflow." I had to implement the deserialization in dll's load()-method cause it worked even worse with Serialize(...) used in documentation/examples I use Visual Studio 2008 and boost1.38 by the way. I "solved" this problem by not de/serializing the next pointer(for linked list) and constructed(set the next pointers) the list with own hands while deserializing. But thats possible only cause of simple structure of linked list and it will not work with more complex structures. Any ideas what I'm doing wrong in code I pasted above and how I get it work? And is serialization of tree-like data structures even possible with dlls involved? Thank you! ps: BOOST_CLASS_EXPORT_GUID() macro which is now commented away caused some weird linking errors. -- View this message in context: http://www.nabble.com/Problems-with-De-Serializing-data-structures-with-dll-... Sent from the Boost - Users mailing list archive at Nabble.com.
cute example. This has nothing to do with dlls etc. The problem is actually quite simple: //The serialization itself oa.register_type(static_cast<BaseImplementation *>(tmp)); for(int i=0; i < 114; i++) { oa << tmp; tmp = tmp->getNext(); //oa << *tmp; } //oa << first; a) Your data structure is a loooong linked list of pointers. b) Serialization of the first pointer in the list is going to recursively serialize all the objects in the list. c) subsequent serialization of any previously serialized ones will be optimized away. If you want to do this in this way, The obvious way would be to make sure there's enough stack space. Remember that serialization is a recurrsive process. If you really need to serialize an arbitrarily long linked list, you'll have to think of another way. You might want to look at the implementation of serialization for std::list in the library which serializes the data but reconstructs the links on load (using push back) rather than serializating them. This was done not to avoid this problem but rather because it made use of the public interface of std::list. Robert Ramey P.S. Usage of export often cause "weird linking errors" do to the need to explicitly instantiate code not referred to by name. One then has to invest some effort in thinking about getting stuff serialized. This conflicts with the goal of the library to permit usage of serialization by those who have a lot of other stuff to do. RR.
Robert Ramey wrote:
This has nothing to do with dlls etc. The problem is actually quite simple: ...
a) Your data structure is a loooong linked list of pointers. b) Serialization of the first pointer in the list is going to recursively serialize all the objects in the list. c) subsequent serialization of any previously serialized ones will be optimized away.
If you want to do this in this way, The obvious way would be to make sure there's enough stack space. Remember that serialization is a recurrsive process. If you really need to serialize an arbitrarily long linked list, you'll have to think of another way. You might want to look at the implementation of serialization for std::list in the library which serializes the data but reconstructs the links on load (using push back) rather than serializating them. This was done not to avoid this problem but rather because it made use of the public interface of std::list.
Robert Ramey
P.S. Usage of export often cause "weird linking errors" do to the need to explicitly instantiate code not referred to by name. One then has to invest some effort in thinking about getting stuff serialized. This conflicts with the goal of the library to permit usage of serialization by those who have a lot of other stuff to do.
RR.
Thanks for your reply Robert! I thought a while the problem could be call stack(as the error message says :), but actually couldn't believe it will get full with 114 cells(with this platform). The reason I mentioned dlls was that I thought the problem could something to do with memory management of dlls( I'm quite noob with dlls also). But as I mentioned before linked list is not the issue in this. I used it as a simple test code. The purpose is to use the de/serialization to save/load the data content of our software and list(neither any other stl-container) is not suitable for this purpose. It is more tree-like structure which doesn't fit in easily in stl-containers. The amount of data can be really large and some meta data is needed if we use the solution which saves only the nodes/leafs of the tree and while loading set the cell's pointers right. Reconstruction will be difficult with meta data also. So it seems that the only solution to do the boost::de/serialization of large/huge self-made data structures easily is increase the size of call stack? Ps: Sorry about the "weird linking errors". I thought export could help me with this one and tried to give a hint I've also tried export. -- View this message in context: http://www.nabble.com/Problems-with-De-Serializing-data-structures-with-dll-... Sent from the Boost - Users mailing list archive at Nabble.com.
As I said - two solutions a) real simple - just increase the call stack. But this would mean that there might be problems with structures with indirection which goes too deep. b) use a system similar to the serialization of std::list which serializes only the data but rebuilds the links when the data is loaded. Robert Ramey Grimm wrote:
Robert Ramey wrote:
This has nothing to do with dlls etc. The problem is actually quite simple: ...
a) Your data structure is a loooong linked list of pointers. b) Serialization of the first pointer in the list is going to recursively serialize all the objects in the list. c) subsequent serialization of any previously serialized ones will be optimized away.
If you want to do this in this way, The obvious way would be to make sure there's enough stack space. Remember that serialization is a recurrsive process. If you really need to serialize an arbitrarily long linked list, you'll have to think of another way. You might want to look at the implementation of serialization for std::list in the library which serializes the data but reconstructs the links on load (using push back) rather than serializating them. This was done not to avoid this problem but rather because it made use of the public interface of std::list.
Robert Ramey
P.S. Usage of export often cause "weird linking errors" do to the need to explicitly instantiate code not referred to by name. One then has to invest some effort in thinking about getting stuff serialized. This conflicts with the goal of the library to permit usage of serialization by those who have a lot of other stuff to do.
RR.
Thanks for your reply Robert!
I thought a while the problem could be call stack(as the error message says :), but actually couldn't believe it will get full with 114 cells(with this platform). The reason I mentioned dlls was that I thought the problem could something to do with memory management of dlls( I'm quite noob with dlls also).
But as I mentioned before linked list is not the issue in this. I used it as a simple test code.
The purpose is to use the de/serialization to save/load the data content of our software and list(neither any other stl-container) is not suitable for this purpose. It is more tree-like structure which doesn't fit in easily in stl-containers. The amount of data can be really large and some meta data is needed if we use the solution which saves only the nodes/leafs of the tree and while loading set the cell's pointers right. Reconstruction will be difficult with meta data also.
So it seems that the only solution to do the boost::de/serialization of large/huge self-made data structures easily is increase the size of call stack?
Ps: Sorry about the "weird linking errors". I thought export could help me with this one and tried to give a hint I've also tried export.
participants (2)
-
Grimm
-
Robert Ramey