boost.MPI dynamic data structure error
Hi , I need to design a data structure to transfer data between nodes by boost.MPI. Some elements of the the structure has dynamic size. For example, typedef struct{ double data1;vector<double> dataVec; } myDataType; The size of the dataVec depends on some intermidiate computing results. If I only declear it as the above myDataType, I think, only a pointer is transfered. When the data receiver try to access the elements of vector<double> dataVec, it got segmentation fault error. But, I also need to use the myDataType to declear other data structures. such as vector<myDataType> newDataVec; I cannot declear myDataType in a function , sucjh as main(), I got errors: main.cpp:200: error: main(int, char**)::myDataType; uses local type main(int, char**)::myDataType; Any help is really appreciated. thanks Jack Oct. 19 2010
On 19 Oct 2010, at 13:39, Jack Bryan wrote:
Hi ,
I need to design a data structure to transfer data between nodes by boost.MPI.
Some elements of the the structure has dynamic size.
For example,
typedef struct{
double data1; vector<double> dataVec;
} myDataType;
The size of the dataVec depends on some intermidiate computing results.
If I only declear it as the above myDataType, I think, only a pointer is transfered.
When the data receiver try to access the elements of vector<double> dataVec, it got segmentation fault error.
But, I also need to use the myDataType to declear other data structures.
such as vector<myDataType> newDataVec;
Hi Jack, All you need to do is to define a serialize() function for your data structure, so that it can be serialized by Boost.Serialization. Then you can transfer the datastructure with Boost.MPI Matthias
thanks I have two problems so that I cannot use boost.MPI. I need to transfer data very frequently, and calling boost Serialization will increase overhead by system call. Because every time I call mpi_send, the Serialization() needs to be called. This will reduce bandwith on MPI systems. Does boost.MPI support dynamic-size vector Serialization ? For example, I declare a class myDataS { double data1;vector<double> dataVec; } myData; This class is visible to all processors. In processor 1, I initialize dataVec(mysize); // mysize is not visible to other processors except processor 1. I use world.send() to send myData to processor 2 by Serialization() in the class myDataS . Does the processor2 knows that how large the dataVec is ? and in processor2, in world.recv(), if I use myData to receive the data from processor 1, how to make sure that all elements in the vector have been received ? Any help is appreciated. Jack Oct. 19 2010 From: troyer@phys.ethz.ch Date: Tue, 19 Oct 2010 17:12:33 -0700 To: boost-users@lists.boost.org Subject: Re: [Boost-users] boost.MPI dynamic data structure error On 19 Oct 2010, at 13:39, Jack Bryan wrote:Hi , I need to design a data structure to transfer data between nodes by boost.MPI. Some elements of the the structure has dynamic size. For example, typedef struct{ double data1;vector<double> dataVec; } myDataType; The size of the dataVec depends on some intermidiate computing results. If I only declear it as the above myDataType, I think, only a pointer is transfered. When the data receiver try to access the elements of vector<double> dataVec, it got segentation fault error. But, I also need to use the myDataType to declear other data structures. such as vector<myDataType> newDataVec; Hi Jack, All you need to do is to define a serialize() function for your data structure, so that it can be serialized by Boost.Serialization. Then you can transfer the datastructure with Boost.MPI Matthias _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 19 Oct 2010, at 17:39, Jack Bryan wrote:
thanks
I have two problems so that I cannot use boost.MPI.
Neither is a problem, as you'll see below.
I need to transfer data very frequently, and calling boost Serialization will increase overhead by system call.
Because every time I call mpi_send, the Serialization() needs to be called.
This will reduce bandwith on MPI systems.
Boost.MPI has been designed to deal with this. Look at the skeleton&content mechanism. If you're sending the same datastructure many times there will be no cost beyond the first send. See also Dave's nice article here: http://daveabrahams.com/2010/09/03/whats-so-cool-about-boost-mpi/
Does boost.MPI support dynamic-size vector Serialization ?
Yes, definitely. It was designed for that and I doubt that you will see any performance penalty compared to a hand-written solution.
For example,
I declare a
class myDataS {
double data1; vector<double> dataVec;
} myData;
This class is visible to all processors.
In processor 1, I initialize dataVec(mysize); // mysize is not visible to other processors except processor 1.
I use world.send() to send myData to processor 2 by Serialization() in the class myDataS .
Does the processor2 knows that how large the dataVec is ? and in processor2, in world.recv(), if I use myData to receive the data from processor 1, how to make sure that all elements in the vector have been received ?
Yes, it does. Matthias
participants (2)
-
Jack Bryan -
Matthias Troyer