On Thu, Sep 17, 2009 at 15:10, Peter Soetens <peter.soetens@gmail.com> wrote:
Hi,
I'm trying to find out if boost::serialization can be used in real-time applications to stream data into a fifo to another process. It is mandatory that no memory allocations happen during the serialization. I tested this with a std::vector<double> of 10 elements in combination with the boost::iostreams library.
My assumptions about the cause of the memory allocation were wrong. I could trace it to this point: #0 0x00007fdcbf4ed9a0 in operator new () from /usr/lib/libstdc++.so.6 #1 0x00007fdcbf76c984 in std::_Rb_tree<boost::archive::detail::basic_oarchive_impl::cobject_type, boost::archive::detail::basic_oarchive_impl::cobject_type, std::_Identity<boost::archive::detail::basic_oarchive_impl::cobject_type>, std: :less<boost::archive::detail::basic_oarchive_impl::cobject_type>, std::allocator<boost::archive::detail::basic_oarchive_impl::cobject_type>
::_M_insert_ (this=0x94b3b8, __x=0x0, __p=0x94b3c0, __v=@0x7fffc7bd0620) at /usr/include/c++/4.3/ext/new_allocator.h:92 #2 0x00007fdcbf76c734 in boost::archive::detail::basic_oarchive::save_object (this=0x7fffc7bd0830, x=0x7fffc7bd0860, bos=@0x611550) at /usr/include/c++/4.3/bits/stl_tree.h:1148 #3 0x000000000040a5b5 in boost::archive::detail::save_non_pointer_type<boost::archive::binary_oarchive, std::vector<double, std::allocator<double> > >::save_standard::invoke (ar=@0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/oserializer.hpp:231 #4 0x000000000040a5d4 in boost::archive::detail::save_non_pointer_type<boost::archive::binary_oarchive, std::vector<double, std::allocator<double> > ::save_conditional::invoke (ar=@0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/oserializer.hpp:245 #5 0x000000000040a5f3 in boost::archive::detail::save_non_pointer_type<boost::archive::binary_oarchive, std::vector<double, std::allocator<double> > >::invoke (ar=@0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/oserializer.hpp:294 #6 0x000000000040a612 in boost::archive::save<boost::archive::binary_oarchive, std::vector<double, std::allocator<double> > > (ar=@0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/oserializer.hpp:506 #7 0x000000000040a63c in boost::archive::detail::common_oarchive<boost::archive::binary_oarchive>::save_override<std::vector<double, std::allocator<double> > const> (this=0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/common_oarchive.hpp:64 #8 0x000000000040a661 in boost::archive::basic_binary_oarchive<boost::archive::binary_oarchive>::save_override<std::vector<double, std::allocator<double> > > (this=0x7fffc7bd0830, t=@0x7fffc7bd0860, version=0) at /usr/include/boost/archive/basic_binary_oarchive.hpp:63 #9 0x000000000040a689 in boost::archive::binary_oarchive_impl<boost::archive::binary_oarchive, char, std::char_traits<char> >::save_override<std::vector<double, std::allocator<double> > > (this=0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/binary_oarchive_impl.hpp:45 #10 0x000000000040a6b6 in boost::archive::detail::interface_oarchive<boost::archive::binary_oarchive>::operator<< <std::vector<double, std::allocator<double> > > (this=0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/interface_oarchive.hpp:64 #11 0x0000000000404a65 in main (argc=1, argv=0x7fffc7bd0f08) at serialize-rt-init.cpp:17
Which looks like std::map is being used to store class data (or so). It looks like it's going to be much harder to work around this than I assumed ? Peter
The aim is that all memory allocations happen during construction of the archive object, while the serialisation itself causes none.
<code> #include <boost/archive/binary_oarchive.hpp> #include <boost/serialization/vector.hpp> #include <boost/iostreams/stream.hpp> #include <boost/iostreams/device/array.hpp>
namespace io = boost::iostreams; int main(int argc, char *argv[]) { vector<double> d(10, 1.1); char sink[1000]; memset( sink, 0, 1000); io::stream<io::array_sink> out(sink,1000);
boost::archive::binary_oarchive oa(out); //oa << d; // should not allocate return 0; } </code>
The setup code does 10 memory allocations according to valgrind: ==12995== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 1) ==12995== malloc/free: in use at exit: 0 bytes in 0 blocks. ==12995== malloc/free: 10 allocs, 10 frees, 913 bytes allocated. ==12995== For counts of detected errors, rerun with: -v ==12995== All heap blocks were freed -- no leaks are possible.
If we uncomment 'oa << d' we get 2 more (unwanted) allocations: ==13010== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 1) ==13010== malloc/free: in use at exit: 0 bytes in 0 blocks. ==13010== malloc/free: 12 allocs, 12 frees, 1,001 bytes allocated. ==13010== For counts of detected errors, rerun with: -v ==13010== All heap blocks were freed -- no leaks are possible
I'm guessing that the 2 allocations in the serialisation path come from a temporay std::string object, when writing the 'serialization::archive' string into the archive. Wouldn't it be possible to rewrite this library code as such that there are no allocations / strings created ? A similar pattern is observed for deserializing: 2 allocs in the serialisation path.
Peter
PS: I tried to find out who defines the macro BOOST_ARCHIVE_SIGNATURE() but I couldn't find the definition/#define !?