[serialization] In-memory archive?
Is there an easy way to serialize into a in-memory archive so that an object can be, say, transferred over a network? Of course one can use a disk file as the intermediate step but having some in-memory archive should help here. Actually, as a related question, is there a way to serialize into and deserialize from some map-like in-memory archive? Just to illustrate the idea better, say we have a Point class. class Point { double m_x, m_y; public: Point(double x, double y): m_x(x), m_y(y) {} }; We can use the constructor Point(1.0, 2.0) to get an object. But how about if we want to construct a Point object from a std::map that has "x => 1.0; y=> 2.0" as its key-value pairs? I guess my example is too simple so that deserializing from a map is not necessary. But consider deserializing from a database, where one natural way to present the information is not a long binary string but a list of key-value pairs. Or consider a generic factory that takes in such a map and gives out a constructed object. Note that a map usually doesn't keep entries in order so this may add extra burden to the serializing code. Please let me know if what I am looking for is too crazy. Any input is appreciated.
On Oct 16, 2008, at 11:05 PM, Ling Li wrote:
Is there an easy way to serialize into a in-memory archive so that an object can be, say, transferred over a network? Of course one can use a disk file as the intermediate step but having some in-memory archive should help here.
You can use any archive type with an in-memory streambuffer. Boost.IOstreams can be used to create these streambuffers. Matthias
Matthias Troyer wrote:
On Oct 16, 2008, at 11:05 PM, Ling Li wrote:
Is there an easy way to serialize into a in-memory archive so that an object can be, say, transferred over a network? Of course one can use a disk file as the intermediate step but having some in-memory archive should help here.
You can use any archive type with an in-memory streambuffer. Boost.IOstreams can be used to create these streambuffers.
Or just std::istringstream/std::ostringstream. Jeff
Thanks! Using streambuffer or std::istringstream/std::ostringstream is indeed very convenient. Has anyone thought about introducing default values in serialization to save the storage size (or the number of bits tranferred over network)? I hacked NVP and make_nvp to support this as an optional feature, and it works fine for me. If that sounds something might be useful to others, I can paste the patch here. On Fri, Oct 17, 2008 at 11:13 AM, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:
Matthias Troyer wrote:
On Oct 16, 2008, at 11:05 PM, Ling Li wrote:
Is there an easy way to serialize into a in-memory archive so that an object can be, say, transferred over a network? Of course one can use a disk file as the intermediate step but having some in-memory archive should help here.
You can use any archive type with an in-memory streambuffer. Boost.IOstreams can be used to create these streambuffers.
Or just std::istringstream/std::ostringstream.
Jeff
Ling Li wrote:
Thanks! Using streambuffer or std::istringstream/std::ostringstream is indeed very convenient.
Has anyone thought about introducing default values in serialization to save the storage size (or the number of bits tranferred over network)? I hacked NVP and make_nvp to support this as an optional feature, and it works fine for me. If that sounds something might be useful to others, I can paste the patch here.
You can get these numbers by making a filter device that counts bytes, (iirc there is an example in iostreams) or by serializing to, say, a vector<char> via via one of the iostreams push_back devices and getting the container's size after the archive is closed. Serializing to a vector is also convenient if you want to, say, use boost::crc to take and transmit checksums with the data. -t
On Fri, Oct 17, 2008 at 12:42 PM, troy d. straszheim <troy@resophonic.com> wrote:
Ling Li wrote:
Thanks! Using streambuffer or std::istringstream/std::ostringstream is indeed very convenient.
Has anyone thought about introducing default values in serialization to save the storage size (or the number of bits tranferred over network)? I hacked NVP and make_nvp to support this as an optional feature, and it works fine for me. If that sounds something might be useful to others, I can paste the patch here.
You can get these numbers by making a filter device that counts bytes, (iirc there is an example in iostreams) or by serializing to, say, a vector<char> via via one of the iostreams push_back devices and getting the container's size after the archive is closed. Serializing to a vector is also convenient if you want to, say, use boost::crc to take and transmit checksums with the data.
-t
Sorry that I didn't make myself clear. I meant to reduce the archive size (not count the size). For example, in demo_gps.hpp, instead of ar & boost::serialization::make_nvp("hour", hour) one might want to use ar & boost::serialization::make_nvp("hour", hour, 11) so that if the hour is 11, it is not serialized in the archive. During deserialization, if the hour is missing in the archive, 11 will be the default value. I have to say this can only be applied to archives with names, e.g., XML. And one can't use default values for elements in a vector unless we give each element a name (e.g., the index). So compared to using the binary archive, it might not be a save in archive size.
Ling Li wrote:
On Fri, Oct 17, 2008 at 12:42 PM, troy d. straszheim <troy@resophonic.com> wrote:
Ling Li wrote:
Thanks! Using streambuffer or std::istringstream/std::ostringstream is indeed very convenient.
Has anyone thought about introducing default values in serialization ...
Sorry that I didn't make myself clear. I meant to reduce the archive size (not count the size). For example, in demo_gps.hpp, instead of ar & boost::serialization::make_nvp("hour", hour) one might want to use ar & boost::serialization::make_nvp("hour", hour, 11) so that if the hour is 11, it is not serialized in the archive. During deserialization, if the hour is missing in the archive, 11 will be the default value.
If you want more compact archives don't use xml. Use either text or portable_binary(or plain binary for homogenous platforms). Additionally you can use these with boost.iostreams compressed filters. The structure of your class heirarchy drives the output/input of archive data. They only optional aspect is to use the versioning facility. Jeff
participants (4)
-
Jeff Flinn -
Ling Li -
Matthias Troyer -
troy d. straszheim