Serialization of dynamic data

Hi! About the boost::serialization library, what is the best way to handle dynamic data in classes? Any idéas, suggestions or caveats? I didn't manage to find anything about this in the docs. Consider the following sample class: struct Dyn { Dyn(int s) : data(0), size(s) { data = new char[size]; } ~Dyn() { delete data; } char* data; int size; private: friend class boost::serialization::access; template< class Archive > void serialize( Archive &ar, unsigned int ver ) { // -> ar & data; What to do here? ar & size; } }; What to do with the data pointer? Is there any way to determine (in the serialize function) if a serialization or "de-serialization" is taking place? And have different behaviours - either allocating a new char[x] or copying data to the archive? Is there any way to store dynamic data in an archive? Best regards, Anders Sundman

-----Original Message----- From: Anders Sundman [mailto:anders.sundman@optonova.se] Sent: Thursday, February 01, 2007 8:55 AM To: boost-users@lists.boost.org Subject: [Boost-users] Serialization of dynamic data Hi! About the boost::serialization library, what is the best way to handle dynamic data in classes? Any idéas, suggestions or caveats? I didn't manage to find anything about this in the docs. <snip> What to do with the data pointer? Is there any way to determine (in the serialize function) if a serialization or "de-serialization" is taking place? And have different behaviours - either allocating a new char[x] or copying data to the archive? Is there any way to store dynamic data in an archive? Hi Anders, I had to do exactly this same thing in our application. Below is a snip of code from my archivable class. // Serializes this object. friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & m_lId; ar & m_timestamp; ar & m_elements; ar & m_unLength; if(m_unLength > 0) { if (Archive::is_loading::value) { m_pData = malloc(m_unLength); } ar & boost::serialization::make_binary_object(m_pData, m_unLength); } } You class will need to keep track of the size of the dynamic data to properly create the temporary object. This is covered in the serialization documentation under Serialization Wrappers. http://www.boost.org/libs/serialization/doc/wrappers.html#binaryobjects Dave

Great! Thanks a bunch. This is exactly what I would like to do. Perfect. // Anders Sundman Dave Nay skrev:
-----Original Message----- From: Anders Sundman [mailto:anders.sundman@optonova.se] Sent: Thursday, February 01, 2007 8:55 AM To: boost-users@lists.boost.org Subject: [Boost-users] Serialization of dynamic data
Hi!
About the boost::serialization library, what is the best way to handle dynamic data in classes? Any idéas, suggestions or caveats? I didn't manage to find anything about this in the docs.
<snip>
What to do with the data pointer?
Is there any way to determine (in the serialize function) if a serialization or "de-serialization" is taking place? And have different behaviours - either allocating a new char[x] or copying data to the archive?
Is there any way to store dynamic data in an archive?
Hi Anders,
I had to do exactly this same thing in our application. Below is a snip of code from my archivable class.
// Serializes this object. friend class boost::serialization::access;
template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & m_lId; ar & m_timestamp; ar & m_elements; ar & m_unLength;
if(m_unLength > 0) { if (Archive::is_loading::value) { m_pData = malloc(m_unLength); }
ar & boost::serialization::make_binary_object(m_pData, m_unLength); } }
You class will need to keep track of the size of the dynamic data to properly create the temporary object.
This is covered in the serialization documentation under Serialization Wrappers. http://www.boost.org/libs/serialization/doc/wrappers.html#binaryobjects
Dave

Ok, my bad. I've found the docs:
http://www.boost.org/libs/serialization/doc/serialization.html#splitting
But the general question remains, what to do with dynamic data?
It's often the case that I find a T* and a size in classes. I've been
thinking about introducing some kind of convenience functions for this.
Perhaps something along these lines:
template
Hi!
About the boost::serialization library, what is the best way to handle dynamic data in classes? Any idéas, suggestions or caveats? I didn't manage to find anything about this in the docs.
Consider the following sample class:
struct Dyn { Dyn(int s) : data(0), size(s) { data = new char[size]; }
~Dyn() { delete data; }
char* data; int size;
private: friend class boost::serialization::access; template< class Archive > void serialize( Archive &ar, unsigned int ver ) { // -> ar & data; What to do here? ar & size; } };
What to do with the data pointer?
Is there any way to determine (in the serialize function) if a serialization or "de-serialization" is taking place? And have different behaviours - either allocating a new char[x] or copying data to the archive?
Is there any way to store dynamic data in an archive?
Best regards, Anders Sundman

Your solution below would likely work. These days I would expect find
the T* / size pair replaced with something like std::vector
Ok, my bad. I've found the docs:
http://www.boost.org/libs/serialization/doc/serialization.html#splitting
But the general question remains, what to do with dynamic data?
It's often the case that I find a T* and a size in classes. I've been thinking about introducing some kind of convenience functions for this. Perhaps something along these lines:
template
void saveBuf(Archive & ar, const T * buf, size_t size) { bool isNull = (buf == 0); ar << size; ar << isNull; if (!isNull) ar.save_binary(buf, size * sizeof(T)); } template
void loadBuf(Archive & ar, T *& buf, size_t & size) { bool isNull; ar >> size; ar >> isNull; if (buf != 0) { delete buf; buf = 0; }
if(!isNull) { buf = new T[size]; ar.load_binary(buf, size * sizeof(T)); } else { buf = 0; } }
What do you thing about that? Any obvious problems? Are there perhaps already something like this in the library?
// Anders Sundman

Thank you for your answer.
Unfortunately, I don't write the classes I have to serialize :)
I've managed (with the help of Dave Nay) to distill my helper function
to the more compact:
template
Your solution below would likely work. These days I would expect find the T* / size pair replaced with something like std::vector
tarray in which case you could have just used: ar & tarray
But you're solution is a natural way to proceed under the circumstances.
BUT - it presumes that T can be serialized as a binary which is not portable and not generally true.
A general solution would be to use a loop which
... ar & t[i]
For examples, look at the implementation of serialization for collections as included in the library.
Robert Ramey
Anders Sundman wrote:
Ok, my bad. I've found the docs:
http://www.boost.org/libs/serialization/doc/serialization.html#splitting
But the general question remains, what to do with dynamic data?
It's often the case that I find a T* and a size in classes. I've been thinking about introducing some kind of convenience functions for this. Perhaps something along these lines:
template
void saveBuf(Archive & ar, const T * buf, size_t size) { bool isNull = (buf == 0); ar << size; ar << isNull; if (!isNull) ar.save_binary(buf, size * sizeof(T)); } template
void loadBuf(Archive & ar, T *& buf, size_t & size) { bool isNull; ar >> size; ar >> isNull; if (buf != 0) { delete buf; buf = 0; }
if(!isNull) { buf = new T[size]; ar.load_binary(buf, size * sizeof(T)); } else { buf = 0; } }
What do you thing about that? Any obvious problems? Are there perhaps already something like this in the library?
// Anders Sundman

Robert Ramey
Your solution below would likely work. These days I would expect find the T* / size pair replaced with something like std::vector
tarray in which case you could have just used: ar & tarray
Is there a possibilty to say "store as binary data" even for xml archives? Sometimes the <item></item> overhead is just too big and I'd prefer the archive to include a CDATA section in that case. All other items should remain human readable, UI like this: ar & stored_as_binary_data(tarray); This would be very useful for std::vector<double> and friends ... Markus

Just wrap it as a "binary_object". Use: ar & NVP(binary_object(tarray, sizeof(tarray)); Robert Ramey Markus Werle wrote:
Robert Ramey
writes: Your solution below would likely work. These days I would expect find the T* / size pair replaced with something like std::vector
tarray in which case you could have just used: ar & tarray
Is there a possibilty to say "store as binary data" even for xml archives? Sometimes the <item></item> overhead is just too big and I'd prefer the archive to include a CDATA section in that case. All other items should remain human readable, UI like this:
ar & stored_as_binary_data(tarray);
This would be very useful for std::vector<double> and friends ...
Markus

A new, slightly different question on the same subject: How to treat data with special "allocator" functions? Ex: In pseudo code, I would like to do something like this: // Serialize Fu* bar = allocFu(); boost::archive::text_oarchive oa(stream, flags); oa << bar; destroyFu(bar); // De-serialize Fu* foo = 0; boost::archive::text_iarchive ia(stream, flags); ia >> foo; // Use foo destroyFu(foo); But how do I create the load/save functions for this: template<class Archive> void save(Archive & ar, const Fu* & t, unsigned int version) { ... } template<class Archive> void load(Archive & ar, Fu* & t, unsigned int version) { ... (run t = allocFu(), among other things) ... } BOOST_SERIALIZATION_SPLIT_FREE(Fu*) But the compiler trips on a static assertion with the comment: if your program traps here, it indicates taht your doing one of the following: a) serializing an object of a type marked "track_never" through a pointer. b) saving an non-const object of a type not markd "track_never) Either of these conditions may be an indicator of an error usage of the serialization library and should be double checked. See documentation on object tracking. I don't really understand the error message, and I can't seem to figure out whats wrong with the code. Any idéas anyone? // Anders Sundman Anders Sundman skrev:
Hi!
About the boost::serialization library, what is the best way to handle dynamic data in classes? Any idéas, suggestions or caveats? I didn't manage to find anything about this in the docs.
Consider the following sample class:
struct Dyn { Dyn(int s) : data(0), size(s) { data = new char[size]; }
~Dyn() { delete data; }
char* data; int size;
private: friend class boost::serialization::access; template< class Archive > void serialize( Archive &ar, unsigned int ver ) { // -> ar & data; What to do here? ar & size; } };
What to do with the data pointer?
Is there any way to determine (in the serialize function) if a serialization or "de-serialization" is taking place? And have different behaviours - either allocating a new char[x] or copying data to the archive?
Is there any way to store dynamic data in an archive?
Best regards, Anders Sundman

This turned out to be a const issue easily solved by reading the docs :) (see: rationale section). // Anders Anders Sundman skrev:
A new, slightly different question on the same subject: How to treat data with special "allocator" functions?
Ex:
In pseudo code, I would like to do something like this:
// Serialize Fu* bar = allocFu(); boost::archive::text_oarchive oa(stream, flags); oa << bar; destroyFu(bar);
// De-serialize Fu* foo = 0; boost::archive::text_iarchive ia(stream, flags); ia >> foo; // Use foo destroyFu(foo);
But how do I create the load/save functions for this:
template<class Archive> void save(Archive & ar, const Fu* & t, unsigned int version) { ... } template<class Archive> void load(Archive & ar, Fu* & t, unsigned int version) { ... (run t = allocFu(), among other things) ... } BOOST_SERIALIZATION_SPLIT_FREE(Fu*)
But the compiler trips on a static assertion with the comment:
if your program traps here, it indicates taht your doing one of the following: a) serializing an object of a type marked "track_never" through a pointer. b) saving an non-const object of a type not markd "track_never) Either of these conditions may be an indicator of an error usage of the serialization library and should be double checked. See documentation on object tracking.
I don't really understand the error message, and I can't seem to figure out whats wrong with the code. Any idéas anyone?
// Anders Sundman
Anders Sundman skrev:
Hi!
About the boost::serialization library, what is the best way to handle dynamic data in classes? Any idéas, suggestions or caveats? I didn't manage to find anything about this in the docs.
Consider the following sample class:
struct Dyn { Dyn(int s) : data(0), size(s) { data = new char[size]; }
~Dyn() { delete data; }
char* data; int size;
private: friend class boost::serialization::access; template< class Archive > void serialize( Archive &ar, unsigned int ver ) { // -> ar & data; What to do here? ar & size; } };
What to do with the data pointer?
Is there any way to determine (in the serialize function) if a serialization or "de-serialization" is taking place? And have different behaviours - either allocating a new char[x] or copying data to the archive?
Is there any way to store dynamic data in an archive?
Best regards, Anders Sundman
participants (4)
-
Anders Sundman
-
Dave Nay
-
Markus Werle
-
Robert Ramey