Boost Serialization: Serializing large objects
We are trying to text serialize a 700 MB object. We are getting an archive_exception: input stream error. This is on a 32-bit QNX system. * Is there a limit to the size of the object that can be serialized? * Is there anything we can do to fix or work around this? Here is the code we are using: std::stringstream ss_; boost::archive::text_oarchive oa(ss_, boost::archive::no_header); oa << &object; Adlai
On 10/9/17 4:21 PM, Adlai Shawareb via Boost wrote:
Thanks for your help. When serializing into a stringstream, how does Boost add or allocate memory to the stringstream?
Boost doesn't do it. std::stringstream does it. It's not too hard to make your own archive which is just a giant memory buffer. I think there's an example in the documentation which does this. But I might be wrong. Robert Ramey
Sorry Robert I wasn't clear. What I meant to ask was which calls to std::stringstream does Boost use to add memory to the stringstream? I looked at the Serialization Tutorial and didn't see an example of making my own archive. If you are inclined, I would appreciate an example. Adlai
On 10/10/17 11:29 AM, Adlai Shawareb via Boost wrote:
I'm sorry, I can't give short answers to these questions. You'll have to study the documentation. Robert Ramey
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
You say the object is 700 megs. Is that 700 megs in memory or in text? The text representation is almost certainly bigger than the in memory representation so you might just be running out of memory. Alternatively, you could just be running out of address space if stringstream requires a contiguous buffer. That's an implementation detail, but I would not be surprised if stringstream allocates a contiguous buffer, in which case you almost certainly would be bumping into a memory fragmentation issue due to the 32bit address space. If you serialize directly to a file instead of stringstream, as Robert suggested, the problem might go away. -- chris
Thanks for responding Chris. The unserialized data is 700 megs. You are correct, the text representation is closer to 1.2 GB. We are using extended addressing on QNX, so I haven't focused on the 32bit address space / fragmentation issue, but it can't be ruled out. The contiguous buffer issue seems likely. The questions I was asking of Robert were to look for something that we could take to QNX to get some feedback. Adlai ________________________________
On 05-10-17 19:41, Adlai Shawareb via Boost wrote:
If you can't write directly to a file, just use `back_insert_device` with a properly reserved container (std::string or std::vector<char> for example): http://www.boost.org/doc/libs/1_65_1/libs/iostreams/doc/classes/back_inserte...
participants (4)
-
Adlai Shawareb -
Chris Glover -
Robert Ramey -
Seth