[serialization] serializing dynamic arrays

newer
Simple Boost Lambda problem using...

older
boost::iostreams - user defined...

Ivan Rachev

9 Jul 2005 9 Jul '05

10:56 p.m.

Does anyone have an idea how to serialize dynamic arrays of objects? An example follows at the end. Its weakness shows up when an outside pointer points to an element in the array. The problem is that the type being serialized is T but not T*. Thanks, Ivan #include <fstream> #include #include #include class A { friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) {} }; //IVAN Element always points to the beginning of an array of // 'Size' elements. The array is NOT null-terminated. If // size == 0, Elements may hold an undefined value. User // is responsible for initializing, new-ing and delete-ing. // Assumption: if there is an outside pointer to an object // inside this array, after loading the array // that pointer will hold a different object. template <typename T> struct DynamicArray { int Size; T* Element; friend class boost::serialization::access; template<class Archive> void save(Archive & ar, const unsigned int version) const { ar & Size; for (int i = 0; i < Size; ++i) ar & Element[i]; } template<class Archive> void load(Archive & ar, const unsigned int version) { ar & Size; //TODO: assert(Size >=0); if (Size > 0) { Element = new T[Size]; for (int i = 0; i < Size; ++i) ar & Element[i]; } } BOOST_SERIALIZATION_SPLIT_MEMBER() }; int main() { DynamicArray<A> ArrayOfAs; A* secondElement; std::ifstream ifs("test.txt"); boost::archive::text_iarchive ia(ifs); ia & ArrayOfAs; ia & secondElement; // this guy will point to a copy of the // 2nd element but NOT the 2nd element itself delete [] ArrayOfAs.Element; }

Show replies by date

Robert Ramey

10 Jul 10 Jul

4:42 a.m.

I believe your example should work as written. Where does it fail? Robert Ramey Ivan Rachev wrote:

...

Does anyone have an idea how to serialize dynamic arrays of objects?

An example follows at the end. Its weakness shows up when an outside pointer points to an element in the array. The problem is that the type being serialized is T but not T*.

Thanks, Ivan

#include <fstream>

#include #include #include

class A { friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) {} };

//IVAN Element always points to the beginning of an array of // 'Size' elements. The array is NOT null-terminated. If // size == 0, Elements may hold an undefined value. User // is responsible for initializing, new-ing and delete-ing. // Assumption: if there is an outside pointer to an object // inside this array, after loading the array // that pointer will hold a different object. template <typename T> struct DynamicArray { int Size; T* Element;

friend class boost::serialization::access; template<class Archive> void save(Archive & ar, const unsigned int version) const { ar & Size; for (int i = 0; i < Size; ++i) ar & Element[i]; } template<class Archive> void load(Archive & ar, const unsigned int version) { ar & Size; //TODO: assert(Size >=0); if (Size > 0) { Element = new T[Size]; for (int i = 0; i < Size; ++i) ar & Element[i]; } } BOOST_SERIALIZATION_SPLIT_MEMBER() };

int main() { DynamicArray<A> ArrayOfAs; A* secondElement;

std::ifstream ifs("test.txt"); boost::archive::text_iarchive ia(ifs); ia & ArrayOfAs; ia & secondElement; // this guy will point to a copy of the // 2nd element but NOT the 2nd element itself

delete [] ArrayOfAs.Element; }

Ivan Rachev

2:36 p.m.

Yes, after further understanding of how boost::serialization works I came to the conclusion that the sample works. Thanks, Ivan Robert Ramey wrote:

...

I believe your example should work as written. Where does it fail?

Robert Ramey

Ivan Rachev wrote:

...
Does anyone have an idea how to serialize dynamic arrays of objects?

An example follows at the end. Its weakness shows up when an outside pointer points to an element in the array. The problem is that the type being serialized is T but not T*.

Thanks, Ivan

#include <fstream>

#include #include #include

class A { friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) {} };

//IVAN Element always points to the beginning of an array of // 'Size' elements. The array is NOT null-terminated. If // size == 0, Elements may hold an undefined value. User // is responsible for initializing, new-ing and delete-ing. // Assumption: if there is an outside pointer to an object // inside this array, after loading the array // that pointer will hold a different object. template <typename T> struct DynamicArray { int Size; T* Element;

friend class boost::serialization::access; template<class Archive> void save(Archive & ar, const unsigned int version) const { ar & Size; for (int i = 0; i < Size; ++i) ar & Element[i]; } template<class Archive> void load(Archive & ar, const unsigned int version) { ar & Size; //TODO: assert(Size >=0); if (Size > 0) { Element = new T[Size]; for (int i = 0; i < Size; ++i) ar & Element[i]; } } BOOST_SERIALIZATION_SPLIT_MEMBER() };

int main() { DynamicArray<A> ArrayOfAs; A* secondElement;

std::ifstream ifs("test.txt"); boost::archive::text_iarchive ia(ifs); ia & ArrayOfAs; ia & secondElement; // this guy will point to a copy of the // 2nd element but NOT the 2nd element itself

delete [] ArrayOfAs.Element; }

Martin Ecker

8:21 a.m.

Hi, Ivan Rachev wrote:

...

Does anyone have an idea how to serialize dynamic arrays of objects?

An example follows at the end.

Please post fully working examples. That will make it easier for people to help you. Your example requires a file test.txt that you didn't send along in your e-mail.

...

Its weakness shows up when an outside pointer points to an element in the array. The problem is that the type being serialized is T but not T*.

This should actually work fine. I modified your sample to use a memory buffer instead of a file and added an assert to make sure the deserialized second-element pointer points to the correct element. The assert does not trigger with MSVC 7.1. #include <sstream> #include #include #include #include class A { friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) {} }; //IVAN Element always points to the beginning of an array of // 'Size' elements. The array is NOT null-terminated. If // size == 0, Elements may hold an undefined value. User // is responsible for initializing, new-ing and delete-ing. // Assumption: if there is an outside pointer to an object // inside this array, after loading the array // that pointer will hold a different object. template <typename T> struct DynamicArray { int Size; T* Element; friend class boost::serialization::access; template<class Archive> void save(Archive & ar, const unsigned int version) const { ar & Size; for (int i = 0; i < Size; ++i) ar & Element[i]; } template<class Archive> void load(Archive & ar, const unsigned int version) { ar & Size; //TODO: assert(Size >=0); if (Size > 0) { Element = new T[Size]; for (int i = 0; i < Size; ++i) ar & Element[i]; } } BOOST_SERIALIZATION_SPLIT_MEMBER() }; int main(int argc, char* argv[]) { DynamicArray<A> ArrayOfAs, ArrayOfAs2; ArrayOfAs.Element = new A[5]; ArrayOfAs.Size = 5; A* secondElement = &ArrayOfAs.Element[1]; A* secondElement2; std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa & ArrayOfAs; oa & secondElement; // this guy will point to a copy of the // 2nd element but NOT the 2nd element itself } { boost::archive::text_iarchive ia(stream); ia & ArrayOfAs2; ia & secondElement2; // this guy will point to a copy of the // 2nd element but NOT the 2nd element itself } // does not trigger, so the above should work assert(secondElement2 == &ArrayOfAs2.Element[1]); delete [] ArrayOfAs.Element; delete [] ArrayOfAs2.Element; } Best Regards, Martin

Ivan Rachev

4:49 p.m.

New subject: [serialization] serializing dynamic arrays of integral types

Hi Martin, If class A from the example is replaced by char, how should this situation be handled? Code follows at the end. Thanks, Ivan template <typename T> struct DynamicArray { int Size; T* Element; friend class boost::serialization::access; template<class Archive> void save(Archive & ar, const unsigned int version) const { ar & Size; for (int i = 0; i < Size; ++i) ar & Element[i]; } template<class Archive> void load(Archive & ar, const unsigned int version) { ar & Size; assert(Size >=0); if (Size > 0) { Element = new T[Size]; for (int i = 0; i < Size; ++i) ar & Element[i]; } } BOOST_SERIALIZATION_SPLIT_MEMBER() }; int main(int argc, char* argv[]) { DynamicArray<char> ArrayOfAs, ArrayOfAs2; ArrayOfAs.Element = new char[5]; ArrayOfAs.Size = 5; char* secondElement = &ArrayOfAs.Element[1]; char* secondElement2; std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa & ArrayOfAs; oa & secondElement; } { boost::archive::text_iarchive ia(stream); ia & ArrayOfAs2; ia & secondElement2; } assert(secondElement2 == &ArrayOfAs2.Element[1]); delete [] ArrayOfAs.Element; delete [] ArrayOfAs2.Element; }

Robert Ramey

8:29 p.m.

New subject: [serialization] serializing dynamic arrays ofintegral types

Addresses of primitive types are generally not tracked by default. So replacing A with a char would not give a desired result. This is described in the manual under the section of seriailzation traites/tracking and serialization of pointers. A solution would be to make a "trackable char" using STRONG_TYPE or some other sort of wrapper to distinguish a char from a your own chartype. Note that if you make a few small changes in your program - and are using current CVS version (1.33 - real soon now) you will an error message to alert you to this problem. Here is my updated version of your program. Ivan Rachev wrote:

...

Hi Martin,

If class A from the example is replaced by char, how should this situation be handled? Code follows at the end.

Thanks, Ivan

template <typename T> struct DynamicArray { int Size; T* Element;

friend class boost::serialization::access; template<class Archive> void save(Archive & ar, const unsigned int version) const { ar & Size; for (int i = 0; i < Size; ++i) ar & Element[i]; } template<class Archive> void load(Archive & ar, const unsigned int version) { ar & Size; assert(Size >=0); if (Size > 0) { Element = new T[Size]; for (int i = 0; i < Size; ++i) ar & Element[i]; } } BOOST_SERIALIZATION_SPLIT_MEMBER() };

int main(int argc, char* argv[]) { DynamicArray<char> ArrayOfAs, ArrayOfAs2; ArrayOfAs.Element = new char[5]; ArrayOfAs.Size = 5; char* secondElement = &ArrayOfAs.Element[1]; char* secondElement2;

std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa & ArrayOfAs; oa & secondElement;

}

{ boost::archive::text_iarchive ia(stream); ia & ArrayOfAs2; ia & secondElement2;

}

assert(secondElement2 == &ArrayOfAs2.Element[1]);

delete [] ArrayOfAs.Element; delete [] ArrayOfAs2.Element; }

Robert Ramey

8:32 p.m.

New subject: [serialization] serializing dynamic arrays ofintegral types

int main(int argc, char* argv[]) { DynamicArray<char> ArrayOfAs, ArrayOfAs2; const ArrayOfAs.Element = new char[5]; ArrayOfAs.Size = 5; char* secondElement = &ArrayOfAs.Element[1]; char* secondElement2; std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa << ArrayOfAs; oa << secondElement; // error message here - serializating a non-tracked type // (a primitive) through a pointer. } Robert Ramey

Martin Ecker

8:39 p.m.

New subject: [serialization] serializing dynamic arrays of integral types

Hi, Ivan Rachev wrote:

...

If class A from the example is replaced by char, how should this situation be handled? Code follows at the end.

This is not recommended. Fundamental types are not tracked by default, so it won't work. You'd have to turn on tracking for char, but that is not a good idea, since then _all_ values of type char that are tracked, and this can cause quite a bit of overhead. If you really need a pointer that directly points into an array (and you can't redesign the code), you could wrap the array elements in a different type that can be tracked in your serialization code. All pointers into the array would also need to be wrapped when serialized. Regards, Martin

Ivan Rachev

13 Jul 13 Jul

9:47 p.m.

New subject: [serialization] serializing dynamic arrays of integral types

Martin Ecker wrote:

...

Hi,

Ivan Rachev wrote:

...
If class A from the example is replaced by char, how should this situation be handled? Code follows at the end.

This is not recommended. Fundamental types are not tracked by default, so it won't work. You'd have to turn on tracking for char, but that is not a good idea, since then _all_ values of type char that are tracked, and this can cause quite a bit of overhead.

If you really need a pointer that directly points into an array (and you can't redesign the code), you could wrap the array elements in a different type that can be tracked in your serialization code. All pointers into the array would also need to be wrapped when serialized.

Regards, Martin

before writing my own wrappers I would like to see if the overhead of tracking chars is acceptable for me. I tried putting the following into my cpp but it won't compile: #include BOOST_CLASS_TRACKING(char, boost::serialization::track_selectivly) I guess I need to change the default "implementation_level" for char. What other things do I need to change so as to make a fundamental type trackable? Thanks, Ivan

Robert Ramey

14 Jul 14 Jul

4:21 a.m.

New subject: [serialization] serializing dynamic arrays ofintegral types

Ivan Rachev wrote:

...

Martin Ecker wrote:

...
Hi,

Ivan Rachev wrote:

...
If class A from the example is replaced by char, how should this situation be handled? Code follows at the end.

This is not recommended. Fundamental types are not tracked by default, so it won't work. You'd have to turn on tracking for char, but that is not a good idea, since then _all_ values of type char that are tracked, and this can cause quite a bit of overhead.

If you really need a pointer that directly points into an array (and you can't redesign the code), you could wrap the array elements in a different type that can be tracked in your serialization code. All pointers into the array would also need to be wrapped when serialized.

Regards, Martin

before writing my own wrappers I would like to see if the overhead of tracking chars is acceptable for me. I tried putting the following into my cpp but it won't compile:

#include BOOST_CLASS_TRACKING(char, boost::serialization::track_selectivly)

I guess I need to change the default "implementation_level" for char. What other things do I need to change so as to make a fundamental type trackable?

Thanks, Ivan

Ivan Rachev

2:46 a.m.

New subject: [serialization] serializing dynamic arrays of integral types

Martin Ecker wrote:

...

Hi,

Ivan Rachev wrote:

...
If class A from the example is replaced by char, how should this situation be handled? Code follows at the end.

This is not recommended. Fundamental types are not tracked by default, so it won't work. You'd have to turn on tracking for char, but that is not a good idea, since then _all_ values of type char that are tracked, and this can cause quite a bit of overhead.

If you really need a pointer that directly points into an array (and you can't redesign the code), you could wrap the array elements in a different type that can be tracked in your serialization code. All pointers into the array would also need to be wrapped when serialized.

Regards, Martin

Robert Ramey

4:22 a.m.

New subject: [serialization] serializing dynamic arrays ofintegral types

How did the compile fail? Robert Ramey

...

#include BOOST_CLASS_TRACKING(char, boost::serialization::track_selectivly)

I guess I need to change the default "implementation_level" for char. What other things do I need to change so as to make a fundamental type trackable?

Thanks, Ivan

Ivan Rachev

5:03 a.m.

New subject: [serialization] serializing dynamic arrays ofintegral types

Robert Ramey wrote:

...

How did the compile fail?

Robert Ramey

...
#include BOOST_CLASS_TRACKING(char, boost::serialization::track_selectivly)

I guess I need to change the default "implementation_level" for char. What other things do I need to change so as to make a fundamental type trackable?

Thanks, Ivan

cpp files contains this: #include BOOST_CLASS_TRACKING(char, boost::serialization::track_selectivly) When compiling I get this: error C2027: use of undefined type 'boost::STATIC_ASSERTION_FAILURE<x>' because I believe BOOST_STATIC_ASSERT inside BOOST_CLASS_TRACKING fails. I use boost 1.32 Ivan

Robert Ramey

6:18 a.m.

New subject: [serialization] serializing dynamic arraysofintegral types

Ahhh yes. Basically the STATIC_ASSERT is preventing you from setting tracking for a primitive type. This almost HAS to be an error. Doing this will effect serialization of all char's in your program which is almost certainly what you don't want to do. If you want to do this, You'll have to wrap a char in your own type so its not a primitive anymore. Then it will pass. I'll add a comment to where the STATIC_ASSERT traps so that one will have an explanation when its needed. Robert Ramey Ivan Rachev wrote:

...

Robert Ramey wrote:

...
How did the compile fail?

Robert Ramey

...
#include BOOST_CLASS_TRACKING(char, boost::serialization::track_selectivly)

I guess I need to change the default "implementation_level" for char. What other things do I need to change so as to make a fundamental type trackable?

Thanks, Ivan

cpp files contains this:

#include BOOST_CLASS_TRACKING(char, boost::serialization::track_selectivly)

When compiling I get this: error C2027: use of undefined type 'boost::STATIC_ASSERTION_FAILURE<x>' because I believe BOOST_STATIC_ASSERT inside BOOST_CLASS_TRACKING fails.

I use boost 1.32

Ivan

Ivan Rachev

16 Jul 16 Jul

1:44 a.m.

New subject: [serialization] serializing dynamic -> serializing pointer to element before the array of elemets crashes

Robert Ramey wrote:

...

Ahhh yes. Basically the STATIC_ASSERT is preventing you from setting tracking for a primitive type. This almost HAS to be an error. Doing this will effect serialization of all char's in your program which is almost certainly what you don't want to do. If you want to do this, You'll have to wrap a char in your own type so its not a primitive anymore. Then it will pass.

Let's say the wrapper for char is A (from Martin's example). Then we would have a dynamic array of As. If I try to serialize a pointer pointing to one of these As inside the array, boost::serialization crashes. Example and stack trace follows at the end. Ivan #include <sstream> #include #include #include #include class A { friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) {} }; template <typename T> struct DynamicArray { int Size; T* Element; friend class boost::serialization::access; template<class Archive> void save(Archive & ar, const unsigned int version) const { ar & Size; for (int i = 0; i < Size; ++i) ar & Element[i]; } template<class Archive> void load(Archive & ar, const unsigned int version) { ar & Size; assert( Size >=0 ); if (Size > 0) { Element = new T[Size]; for (int i = 0; i < Size; ++i) ar & Element[i]; } } BOOST_SERIALIZATION_SPLIT_MEMBER() }; int main(int argc, char* argv[]) { DynamicArray<A> ArrayOfAs; ArrayOfAs.Element = new A[5]; ArrayOfAs.Size = 5; A* secondElement = &ArrayOfAs.Element[1]; std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa & secondElement; oa & ArrayOfAs; } delete [] ArrayOfAs.Element; } STACK: 77e9e8bb 77e9e8bb

...

_STL::__stl_debug_engine<bool>::_M_attach(_STL::__owned_list * __l=0x004f7db4, _STL::__owned_link * __c_node=0x00000001) Line 435 C++ _STL::_Atomic_swap(volatile long * __p=0x00000001, long __q=1243908) Line 458 C++ 004f7db4 _CxxThrowException(void * pExceptionObject=0x0012fb14, const _s__ThrowInfo * pThrowInfo=0x004b9c84) C++ boost::throw_exceptionboost::archive::archive_exception(const boost::archive::archive_exception & e={...}) Line 40 C++

boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::detail::basic_oarchive & ar={...}, const void * t=0x004f5951, const boost::archive::detail::basic_oserializer & bos={...}) Line 284 C++ boost::archive::detail::basic_oarchive::save_object(const void * x=0x004f5951, const boost::archive::detail::basic_oserializer & bos={...}) Line 395 C++ boost::archive::detail::save_non_pointer_typeboost::archive::text_oarchive,A::save::invokex(boost::archive::text_oarchive & ar={...}, const A & t={...}) Line 235 C++ boost::archive::detail::save_non_pointer_typeboost::archive::text_oarchive,A::invoke(boost::archive::text_oarchive & ar={...}, const A & t={...}) Line 309 C++ boost::archive::saveboost::archive::text_oarchive,A(boost::archive::text_oarchive & ar={...}, const A & t={...}) Line 551 C++ boost::archive::basic_text_oarchiveboost::archive::text_oarchive::save_override<A>() Line 71 C++ boost::archive::detail::interface_oarchive::operator<<<A>() Line 81 C++ boost::archive::detail::interface_oarchive::operator&<A>() Line 111 C++ DynamicArray<A>::save() Line 26 C++ boost::serialization::access::member_save

...

(boost::archive::text_oarchive & ar={...}, const DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 91 C++

boost::serialization::detail::member_saver

...

::invoke(boost::archive::text_oarchive & ar={...}, const DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 42 C++

boost::serialization::split_member

...

(boost::archive::text_oarchive & ar={...}, DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 67 C++ DynamicArray<A>::serialize() Line 40 C++

boost::serialization::access::serialize

...

(boost::archive::text_oarchive & ar={...}, DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 107 C++

boost::serialization::serialize

...

(boost::archive::text_oarchive & ar={...}, DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 78 C++

boost::serialization::serialize_adl

...

(boost::archive::text_oarchive & ar={...}, DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 121 C++

boost::archive::detail::oserializer

...

::save_object_data(boost::archive::detail::basic_oarchive & ar={...}, const void * x=0x0012ff54) Line 152 C++

boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::detail::basic_oarchive & ar={...}, const void * t=0x0012ff54, const boost::archive::detail::basic_oserializer & bos={...}) Line 257 C++ boost::archive::detail::basic_oarchive::save_object(const void * x=0x0012ff54, const boost::archive::detail::basic_oserializer & bos={...}) Line 395 C++ boost::archive::detail::save_non_pointer_type

...

::save::invokex(boost::archive::text_oarchive & ar={...}, const DynamicArray<A> & t={...}) Line 235 C++

boost::archive::detail::save_non_pointer_type

...

::invoke(boost::archive::text_oarchive & ar={...}, const DynamicArray<A> & t={...}) Line 309 C++ boost::archive::save

boost::archive::basic_text_oarchiveboost::archive::text_oarchive::save_override

...

() Line 71 C++

boost::archive::detail::interface_oarchiveboost::archive::text_oarchive::operator<<

...

() Line 81 C++

boost::archive::detail::interface_oarchiveboost::archive::text_oarchive::operator&

...

(DynamicArray<A> & t={...}) Line 111 C++ main(int argc=1, char * * argv=0x004f7478) Line 55 C++ mainCRTStartup() Line 398 C 77e9ca90

Robert Ramey

4:19 a.m.

New subject: [serialization] serializing dynamic -> serializing pointer to element before the array of elemets crashes

This is a different problem.

...

DynamicArray<A> ArrayOfAs; ArrayOfAs.Element = new A[5]; ArrayOfAs.Size = 5; A* secondElement = &ArrayOfAs.Element[1];

std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa & secondElement; // new object created here oa & ArrayOfAs; // attempt to reload an already loaded object

to new address - throw exception - pointer conflict

...

}

the following should work: oa & ArrayOfAs; // load objects into array oa & secondElement; // load pointer to previouslt created object Robert Ramey Ivan Rachev wrote:

...

Robert Ramey wrote:

...
Ahhh yes. Basically the STATIC_ASSERT is preventing you from setting tracking for a primitive type. This almost HAS to be an error. Doing this will effect serialization of all char's in your program which is almost certainly what you don't want to do. If you want to do this, You'll have to wrap a char in your own type so its not a primitive anymore. Then it will pass.

Let's say the wrapper for char is A (from Martin's example). Then we would have a dynamic array of As. If I try to serialize a pointer pointing to one of these As inside the array, boost::serialization crashes. Example and stack trace follows at the end.

Ivan

#include <sstream>

#include #include #include #include

class A { friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) {} };

template <typename T> struct DynamicArray { int Size; T* Element;

friend class boost::serialization::access; template<class Archive> void save(Archive & ar, const unsigned int version) const { ar & Size; for (int i = 0; i < Size; ++i) ar & Element[i]; } template<class Archive> void load(Archive & ar, const unsigned int version) { ar & Size; assert( Size >=0 ); if (Size > 0) { Element = new T[Size]; for (int i = 0; i < Size; ++i) ar & Element[i]; } } BOOST_SERIALIZATION_SPLIT_MEMBER() };

int main(int argc, char* argv[]) { DynamicArray<A> ArrayOfAs; ArrayOfAs.Element = new A[5]; ArrayOfAs.Size = 5; A* secondElement = &ArrayOfAs.Element[1];

std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa & secondElement; oa & ArrayOfAs; }

delete [] ArrayOfAs.Element; }

STACK: 77e9e8bb 77e9e8bb

...
_STL::__stl_debug_engine<bool>::_M_attach(_STL::__owned_list * __l=0x004f7db4, _STL::__owned_link * __c_node=0x00000001) Line 435 C++ _STL::_Atomic_swap(volatile long * __p=0x00000001, long __q=1243908) Line 458 C++ 004f7db4 _CxxThrowException(void * pExceptionObject=0x0012fb14, const _s__ThrowInfo * pThrowInfo=0x004b9c84) C++ boost::throw_exceptionboost::archive::archive_exception(const boost::archive::archive_exception & e={...}) Line 40 C++

boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::det ail::basic_oarchive

...

& ar={...}, const void * t=0x004f5951, const boost::archive::detail::basic_oserializer & bos={...}) Line 284 C++ boost::archive::detail::basic_oarchive::save_object(const void * x=0x004f5951, const boost::archive::detail::basic_oserializer & bos={...}) Line 395 C++

boost::archive::detail::save_non_pointer_type::save::invokex(boost::archive::text_oarchive

...

& ar={...}, const A & t={...}) Line 235 C++

boost::archive::detail::save_non_pointer_type::invoke(boost::archive::text_oarchive

...

& ar={...}, const A & t={...}) Line 309 C++

boost::archive::saveboost::archive::text_oarchive,A(boost::archive::text_o archive

...

& ar={...}, const A & t={...}) Line 551 C++

boost::archive::basic_text_oarchiveboost::archive::text_oarchive::save_ove rride<A>()

...

Line 71 C++

boost::archive::detail::interface_oarchiveboost::archive::text_oarchive::o perator<<<A>()

...

Line 81 C++

boost::archive::detail::interface_oarchiveboost::archive::text_oarchive::o perator&<A>()

...

Line 111 C++ DynamicArray<A>::save() Line 26 C++

boost::serialization::access::member_save

...

...
(boost::archive::text_oarchive & ar={...}, const DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 91 C++

boost::serialization::detail::member_saver

...

...
::invoke(boost::archive::text_oarchive & ar={...}, const DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 42 C++

boost::serialization::split_member

...

...
(boost::archive::text_oarchive & ar={...}, DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 67 C++ DynamicArray<A>::serialize() Line 40 C++

boost::serialization::access::serialize

...

...
(boost::archive::text_oarchive & ar={...}, DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 107 C++

boost::serialization::serialize

...

...
(boost::archive::text_oarchive & ar={...}, DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 78 C++

boost::serialization::serialize_adl

...

...
(boost::archive::text_oarchive & ar={...}, DynamicArray<A> & t={...}, const unsigned int file_version=0) Line 121 C++

boost::archive::detail::oserializer

...

...
::save_object_data(boost::archive::detail::basic_oarchive & ar={...}, const void * x=0x0012ff54) Line 152 C++

boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::det ail::basic_oarchive

...

& ar={...}, const void * t=0x0012ff54, const boost::archive::detail::basic_oserializer & bos={...}) Line 257 C++ boost::archive::detail::basic_oarchive::save_object(const void * x=0x0012ff54, const boost::archive::detail::basic_oserializer & bos={...}) Line 395 C++

boost::archive::detail::save_non_pointer_type

...

...
::save::invokex(boost::archive::text_oarchive & ar={...}, const DynamicArray<A> & t={...}) Line 235 C++

boost::archive::detail::save_non_pointer_type

...

...
::invoke(boost::archive::text_oarchive & ar={...}, const DynamicArray<A> & t={...}) Line 309 C++ boost::archive::save

boost::archive::basic_text_oarchiveboost::archive::text_oarchive::save_ove rride

...

...
() Line 71 C++

boost::archive::detail::interface_oarchiveboost::archive::text_oarchive::o perator<<

...

...
() Line 81 C++

boost::archive::detail::interface_oarchiveboost::archive::text_oarchive::o perator&

...

...
(DynamicArray<A> & t={...}) Line 111 C++ main(int argc=1, char * * argv=0x004f7478) Line 55 C++ mainCRTStartup() Line 398 C 77e9ca90

Ivan Rachev

12:10 p.m.

New subject: [serialization] serializing dynamic -> serializing pointer to element before the array of elemets crashes

Robert Ramey wrote:

...

This is a different problem.

...
DynamicArray<A> ArrayOfAs; ArrayOfAs.Element = new A[5]; ArrayOfAs.Size = 5; A* secondElement = &ArrayOfAs.Element[1];

std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa & secondElement; // new object created here oa & ArrayOfAs; // attempt to reload an already loaded object

to new address - throw exception - pointer conflict

...
}

the following should work:

oa & ArrayOfAs; // load objects into array oa & secondElement; // load pointer to previouslt created object

Robert Ramey

yes, it is a different problem. Could you suggest a solution to it? Ivan PS: If the array comes first the problem doesn't appear.

Ivan Rachev

4:17 p.m.

New subject: [serialization] serializing dynamic -> serializing pointer to element before the array of elemets crashes

Robert Ramey wrote:

...

This is a different problem.

...
DynamicArray<A> ArrayOfAs; ArrayOfAs.Element = new A[5]; ArrayOfAs.Size = 5; A* secondElement = &ArrayOfAs.Element[1];

std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa & secondElement; // new object created here oa & ArrayOfAs; // attempt to reload an already loaded object

to new address - throw exception - pointer conflict

...
}

I think over here we have a solution to try. I believe at saving boost::serialization creates a map (object address -> object id) and at loading the lib creates the reverse map (object id -> object address) How inside the archive could I write a function like this: ObjID GetObjectID(void* objectAddress); Usage will be like this: class A { template<class Archive> void save(Archive & ar, const unsigned int version) const { ... ObjID objId = ar.GetObjectID(this); ... } }; Also, after loading is done and the load map is created, how inside the archive could I implement a function like this: void* GetObjectAddress(ObjID objId); Usage will be like this: ... ObjID objId; std::stringstream stream; boost::archive::text_iarchive ia(stream); ia & someObj; ia & objId; void* p = ar.GetObjectAddress(objId); ... Ivan

Robert Ramey

17 Jul 17 Jul

2:20 p.m.

New subject: [serialization] serializing dynamic -> serializing pointer to element before the array of elemets crashes

The problem is is that de-serializing a pointer creates a new object in a new address. Now one comes along an deseriailzed the same object to a fixed address. There is no way to fix this without going back to the begining. This situation is detected though tracking and a "pointer conflict" exception is thrown. The solution is to be sure that if objects are serialized directly and through pointers, the the pointer serialization is done second. Robert Ramey Ivan Rachev wrote:

...

Robert Ramey wrote:

...
This is a different problem.

...
DynamicArray<A> ArrayOfAs; ArrayOfAs.Element = new A[5]; ArrayOfAs.Size = 5; A* secondElement = &ArrayOfAs.Element[1];

std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa & secondElement; // new object created here oa & ArrayOfAs; // attempt to reload an already loaded object

to new address - throw exception - pointer conflict

...
}

I think over here we have a solution to try.

I believe at saving boost::serialization creates a map (object address -> object id) and at loading the lib creates the reverse map (object id -> object address)

How inside the archive could I write a function like this: ObjID GetObjectID(void* objectAddress);

Usage will be like this: class A { template<class Archive> void save(Archive & ar, const unsigned int version) const { ... ObjID objId = ar.GetObjectID(this); ... }

};

Also, after loading is done and the load map is created, how inside the archive could I implement a function like this: void* GetObjectAddress(ObjID objId);

Usage will be like this: ... ObjID objId; std::stringstream stream; boost::archive::text_iarchive ia(stream); ia & someObj; ia & objId; void* p = ar.GetObjectAddress(objId); ...

Ivan

Ivan Rachev

8 Aug 8 Aug

2:54 p.m.

New subject: [serialization] serializing dynamic -> serializing pointer to element before the array of elemets crashes

...

...
The solution is to be sure that if objects are serialized directly and through pointers, the the pointer serialization is done second.

yes, whenever a pointer serialization is done it should be defered and serialized after the direct serialization. My believe is that the lib keeps an object-address-to-object-id mapping (BoostSaveMap) when saving and an object-id-to-object-address mapping (BoostLoadMap) when loading objects from an archive so that one and the same object does not get serialized twice. Based on this assumption there is a solition for the following problematic scenario: struct TChar { char ch; }; template <typename T> struct DynamicArray { int Size; T* Element; }; typedef DynamicArray<TChar> TCharArray; class A { TChar * ptr; } a; class B { TCharArray array; } b; b.array.Element = new TChar[3]; b.array.Size = 3; a.ptr = b.array.Element[1]; ar & a; // at loading, new object created here ar & b; // attempt to reload an already loaded object // to new address - throw exception - pointer conflict patch(ar); // fix to problem that needs implementation For the solution itself, I see the following that needs to happen at saving&loading: When saving, I see 3 steps: 1) whenever TChar* is being saved, adding the TChar* pointer to InternalPtrVector but not sending it to the archive (sending is in step 3). This vector is a mapping of internal pointer (a.ptr) to holder id (ID of a) and contains offset of the pointer (a.ptr) in the object that holds this pointer (a). So for each TChar* in the above example, this vector knows the id of the object that holds the TChar* and its offset in this object. 2) whenever DynamicArray is being saved, adding it to ArrayRangeVector and sending it to the archive. Adding is done by writing the following into the ArrayRangeVector structure: 1.array address (b.array), 2.array size (b.array.Size), 3.array id. This structure knows the range of addresses used by a particular array and the id that boost::serialization gave to the array at saving. 3) after formal serialization (in the patch func) iterate over InternalPtrVector and for each internal pointer there, check if it's within the range of any of the arrays that are stored in ArrayRangeVector. If such an array is found, write the following InternalPtrFileStruct to the archive: Holder ID, (ID of a in the example above) Holder Offset, (offset of a.ptr in a) Array ID, (ID of b.array) Array Offset ( a.ptr - b.array.Element ) The structures used when saving are: BoostLoadMap --------------------------------- | object ID -> object address | --------------------------------- InternalPtrVector ------------------------------------------------- | InternalPtr | Holder ID | Holder Offset | ------------------------------------------------- ArrayRangeVector ----------------------------------------------- | Array Address | Array Size | Array ID | ----------------------------------------------- When loading, I see 1 step: When formal loading is going on, nothing happens to TChar* members. They are written into afterwords. Actual objects (TChar in the case above) are loaded when loading the dynamic array. After formal serialization (ar & a; ar & b; ) is done, TChar* members are initialized in the patch func. What it should do is the following: read each InternalPtrFileStruct. Look up holder object address (&a) in BoostLoadMap by Holder ID (ID of a). Add Holder offset to this address and the result is the address of TChar* in the case above. Now we need to calculate the value for the TChar* member. This is done by getting the address of the array (&b.array.Element) from BoostLoadMap using Array ID (id of b.array). Add Array Offset to this address and the result is the value of the TChar* memeber. So write this value into the address of TChar*. The structure used at loading is : BoostLoadMap --------------------------------- | object address -> object ID | --------------------------------- In order for this to work there needs to be some way of accessing BoostLoadMap and BoostSaveMap from users of boost::serialization (this is where Robert could help). Of course, I believe the better way would be if all this functionality is incorporated into the lib. That way or another there needs to be serialization code similar to the following: // user code below this line --------------------------------------------------------------------------- template <class Archive> void serialize(Archive& ar, A& obj) { saveInternalPtr(obj.ptr, obj, ar); ar & obj.other_members; } template <class Archive> void serialize(Archive& ar, B& obj) { ar & obj.array; } // lib code below this line --------------------------------------------------------------------------- template <class Archive> saveInternalPtr(void* InternalPtr , void* objectAddress, Archive& ar) { if (ar.is_saving) { // get object id by objectAddress from BoostSaveMap InternalPtrVector.insert(InternalPtr, objectID, InternalPtr - objectAddress); } } template void serialize(Archive& ar, DynamicArray<ElemT>& obj) { ar & obj.Size; if (ar.is_saving) { for (int i = 0; i < obj.Size; ++i) ar & obj.Element[i]; // get array id by array Address from BoostSaveMap ArrayRangeVector.insert(&obj, obj.Size, arrayID); } else { assert( obj.Size >=0 ); if (obj.Size > 0) { obj.Element = new ElemT[obj.Size]; for (int i = 0; i < obj.Size; ++i) ar & obj.Element[i]; } else obj.Element = null; } } template <class Archive> void patch(Archive& ar) { ar & InternalPtrVector.size; if (ar.is_saving) // for each internal pointer in InternalPtrVector // for each array in ArrayRangeVector // if internal pointer within [ArrayRangeVector[j].arrayAddress, ArrayRangeVector[j].arrayAddress + ArrayRangeVector[j].arraySize] // ar & InternalPtrVector[i].holderID; // ar & InternalPtrVector[i].holderOffset; // ar & ArrayRangeVector[j].arrayID; // ar & InternalPtrVector[i].InternalPtr - ArrayRangeVector[j].arrayAddress; else // for (int i = 0; i < InternalPtrVector.size; ++i) // ar & holderID; // get holder address by holderID from BoostLoadMap // ar & holderOffset; // void * memberAddress = holderAddress + holderOffset; /* this is address of TChar* in the example above */ // ar & arrayID; // get array address by arrayID from BoostLoadMap // ar & arrayOffset; // int memberValue = arrayAddress + arrayOffset; /* this is value of TChar* in the example above */ // *memberAddress = memberValue; } Ivan Robert Ramey wrote:

...

The problem is is that de-serializing a pointer creates a new object in a new address. Now one comes along an deseriailzed the same object to a fixed address. There is no way to fix this without going back to the begining. This situation is detected though tracking and a "pointer conflict" exception is thrown. The solution is to be sure that if objects are serialized directly and through pointers, the the pointer serialization is done second.

Robert Ramey

Ivan Rachev wrote:

...
Robert Ramey wrote:

...
This is a different problem.

...
DynamicArray<A> ArrayOfAs; ArrayOfAs.Element = new A[5]; ArrayOfAs.Size = 5; A* secondElement = &ArrayOfAs.Element[1];

std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa & secondElement; // new object created here oa & ArrayOfAs; // attempt to reload an already loaded object

to new address - throw exception - pointer conflict

...
}

I think over here we have a solution to try.

I believe at saving boost::serialization creates a map (object address -> object id) and at loading the lib creates the reverse map (object id -> object address)

How inside the archive could I write a function like this: ObjID GetObjectID(void* objectAddress);

Usage will be like this: class A { template<class Archive> void save(Archive & ar, const unsigned int version) const { ... ObjID objId = ar.GetObjectID(this); ... }

};

Also, after loading is done and the load map is created, how inside the archive could I implement a function like this: void* GetObjectAddress(ObjID objId);

Usage will be like this: ... ObjID objId; std::stringstream stream; boost::archive::text_iarchive ia(stream); ia & someObj; ia & objId; void* p = ar.GetObjectAddress(objId); ...

Ivan

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Ivan Rachev

9:04 p.m.

New subject: [serialization] serializing dynamic -> serializing pointer to element before the array of elemets crashes

...

...
The solution is to be sure that if objects are serialized directly and through pointers, the the pointer serialization is done second.

...

The problem is is that de-serializing a pointer creates a new object in a new address. Now one comes along an deseriailzed the same object to a fixed address. There is no way to fix this without going back to the begining. This situation is detected though tracking and a "pointer conflict" exception is thrown. The solution is to be sure that if objects are serialized directly and through pointers, the the pointer serialization is done second.

Robert Ramey

Ivan Rachev wrote:

...
Robert Ramey wrote:

...
This is a different problem.

...
DynamicArray<A> ArrayOfAs; ArrayOfAs.Element = new A[5]; ArrayOfAs.Size = 5; A* secondElement = &ArrayOfAs.Element[1];

std::stringstream stream; { boost::archive::text_oarchive oa(stream); oa & secondElement; // new object created here oa & ArrayOfAs; // attempt to reload an already loaded object

to new address - throw exception - pointer conflict

...
}

I think over here we have a solution to try.

I believe at saving boost::serialization creates a map (object address -> object id) and at loading the lib creates the reverse map (object id -> object address)

How inside the archive could I write a function like this: ObjID GetObjectID(void* objectAddress);

Usage will be like this: class A { template<class Archive> void save(Archive & ar, const unsigned int version) const { ... ObjID objId = ar.GetObjectID(this); ... }

};

Also, after loading is done and the load map is created, how inside the archive could I implement a function like this: void* GetObjectAddress(ObjID objId);

Usage will be like this: ... ObjID objId; std::stringstream stream; boost::archive::text_iarchive ia(stream); ia & someObj; ia & objId; void* p = ar.GetObjectAddress(objId); ...

Ivan

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

7137

Age (days ago)

7167

Last active (days ago)

List overview

Download

20 comments

3 participants

participants (3)

Ivan Rachev
Martin Ecker
Robert Ramey