Boost serialization : Stream error exception
I'm trying to use boost serialization to serialize/deserialize data to and from a mmap'd file. I have my own ostream/istream classes that essentially read/write bytes from a mmap'd file. The process works fine except on some rare occasions (With different objects/number of objects), in which case boost throws a "stream error" exception. Any ideas what could trigger this? A search of the Boost archives indicated other people had the same problem, where the solution was "use binary mode when you open the file"... but I'm not using a file on disk, I'm using a mmap'd file. (As a side note, the mmap file is large enough to hold all my data and does not need to be expanded or remmap'd etc) The OS i'm using is Linux. Thanks for any suggestions... __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On Thursday, August 24, 2006 at 07:33:01 (-0700) smith smithson writes:
I'm trying to use boost serialization to serialize/deserialize data to and from a mmap'd file. I have my own ostream/istream classes that essentially read/write bytes from a mmap'd file. The process works fine except on some rare occasions (With different objects/number of objects), in which case boost throws a "stream error" exception. Any ideas what could trigger this? A search of the Boost archives indicated other people had the same problem, where the solution was "use binary mode when you open the file"... but I'm not using a file on disk, I'm using a mmap'd file. (As a side note, the mmap file is large enough to hold all my data and does not need to be expanded or remmap'd etc)
The OS i'm using is Linux.
Thanks for any suggestions...
I don't use my own mmap classes, I use boost's, under Linux: boost::iostreams::streamboost::iostreams::mapped_file_source ifs(filename); if (compress) { boost::iostreams::filtering_istream in; in.push(boost::iostreams::zlib_decompressor()); in.push(ifs); Archive ia(in); ia >> BOOST_SERIALIZATION_NVP(obj); } else { Archive ia(ifs); ia >> BOOST_SERIALIZATION_NVP(obj); } And it works fine. I do not use it for the write side, as I don't know in advance how large the object will be, etc. I would suggest a few runs with standard streams to see if you still get the error. Boost is independent (should be) of the stream used, so that would be a good test. Bill
Its not clear whether your using binary, text or ? type of archives. Also it would help to know which version of boost you're using. The lastest version of boost in the CVS head (1.35?) use an implementation of binary archives which bypasses streams in favor of direct calls to the stream_buffer. The is measurably faster. Of course I have to be suspicious of you stream i/o re-implementation. Which like all standard library implementations is much trickier to get right than first meets the eye. Other than this I'm not sure what more I can say. I'm sure this doesn't help much - and I'm sorry about that. Robert Ramey smith smithson wrote:
I'm trying to use boost serialization to serialize/deserialize data to and from a mmap'd file. I have my own ostream/istream classes that essentially read/write bytes from a mmap'd file. The process works fine except on some rare occasions (With different objects/number of objects), in which case boost throws a "stream error" exception. Any ideas what could trigger this? A search of the Boost archives indicated other people had the same problem, where the solution was "use binary mode when you open the file"... but I'm not using a file on disk, I'm using a mmap'd file. (As a side note, the mmap file is large enough to hold all my data and does not need to be expanded or remmap'd etc)
The OS i'm using is Linux.
Thanks for any suggestions...
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
I figured out what the problem is; I use a buffered
version of iostream/stream_buf to buffer data from my
mmap'd file and pass it out when requested. For some
reason if the very first character in the buffer is
character 255 (an extended ascii character) in any of
the buffers, I get the stream error exception. Any
suggestions why this might happen? Is this reserved in
some way? I see the issue in both text mode and binary
mode.
I'm using boost 1.32.0.
--- Robert Ramey
Its not clear whether your using binary, text or ? type of archives. Also it would help to know which version of boost you're using. The lastest version of boost in the CVS head (1.35?) use an implementation of binary archives which bypasses streams in favor of direct calls to the stream_buffer. The is measurably faster.
Of course I have to be suspicious of you stream i/o re-implementation. Which like all standard library implementations is much trickier to get right than first meets the eye.
Other than this I'm not sure what more I can say.
I'm sure this doesn't help much - and I'm sorry about that.
Robert Ramey
I'm trying to use boost serialization to serialize/deserialize data to and from a mmap'd file. I have my own ostream/istream classes that essentially read/write bytes from a mmap'd file. The process works fine except on some rare occasions (With different objects/number of objects), in which case boost
a "stream error" exception. Any ideas what could trigger this? A search of the Boost archives indicated other people had the same problem, where the solution was "use binary mode when you open the file"... but I'm not using a file on disk, I'm using a mmap'd file. (As a side note, the mmap file is large enough to hold all my data and does not need to be expanded or remmap'd etc)
The OS i'm using is Linux.
Thanks for any suggestions...
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam
smith smithson wrote: throws protection around
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
It's quite hard to guess what might be the reason for your stream errors without seeing any code or call stacks at exception time. I had some problems in the past with memory mapped files. They happened occasionally with different files on different machines. I don't know your MMF implementation, but when it tries to map the whole file at once, it needs a continuous block of memory and when your are working for some time the memory of your machine might get fragmented so that the allocation fails. If this is the reason for your occasional errors you may want to try to change your MMF implementation so that it only maps a part of the file to memory and shift this 'window' around. Hope that helps. Oliver smith smithson wrote:
I'm trying to use boost serialization to serialize/deserialize data to and from a mmap'd file. I have my own ostream/istream classes that essentially read/write bytes from a mmap'd file. The process works fine except on some rare occasions (With different objects/number of objects), in which case boost throws a "stream error" exception. Any ideas what could trigger this? A search of the Boost archives indicated other people had the same problem, where the solution was "use binary mode when you open the file"... but I'm not using a file on disk, I'm using a mmap'd file. (As a side note, the mmap file is large enough to hold all my data and does not need to be expanded or remmap'd etc)
The OS i'm using is Linux.
Thanks for any suggestions...
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
participants (4)
-
Bill Lear -
Oliver Mutz -
Robert Ramey -
smith smithson