sending large serialized objects over network using MPI
HI, I am involved in a project which requires the transport of large C++ instances over MPI communication channels. I am using boost serialization to pack and unpack the objects into a buffer. The whole buffer is sent in one shot. Of course for large objects this can be problematic. I have two questions... 1) is there any way to tell beforehand how large the buffer needs to be? In other words, can you run serialization in a counting only mode? 2) is there an archive implemented which streams across MPI communication channels (not sure this is the right terminology.)? Seems like it might be useful. This might be a better question for the boost MPI team, though. Thanks, Jon
on Fri Oct 28 2011, Jonathan Bishop
HI,
I am involved in a project which requires the transport of large C++ instances over MPI communication channels. I am using boost serialization to pack and unpack the objects into a buffer. The whole buffer is sent in one shot.
I think you should seriously look into Boost.MPI. It too uses Boost Serialization, but it can often avoid the extra step of packing into a separate buffer.
Of course for large objects this can be problematic. I have two questions...
1) is there any way to tell beforehand how large the buffer needs to be? In other words, can you run serialization in a counting only mode? 2) is there an archive implemented which streams across MPI communication channels (not sure this is the right terminology.)? Seems like it might be useful. This might be a better question for the boost MPI team, though.
Yes, see Boost.MPI. You might also want to read http://daveabrahams.com/2010/09/03/whats-so-cool-about-boost-mpi/ -- Dave Abrahams BoostPro Computing http://www.boostpro.com
participants (2)
-
Dave Abrahams -
Jonathan Bishop