Nick Collier writes:

I running into an issue where an irecv followed by a send results in deadlock. A simple test case, [snip] int other = world.rank() == 0 ? 1 : 0; cout << world.rank() << " irecv from " << other << endl; receipt.request = world.irecv(other, 0, receipt.items); cout << world.rank() << " sending to " << other << endl; world.send(other, 0, msg); [snip]

Does this even work with non-boost MPI? My reading (skimming) of the MPI spec does not generate the implication that synchronous sends can be asynchronously received. See: http://www.mpi-forum.org/docs/mpi21-report/node58.htm#Node58 Note the emphasized part in: The fields in a status object returned by a call to MPI_WAIT, MPI_TEST, or any of the other derived functions *( MPI_{TEST|WAIT}{ALL|SOME|ANY}), where the request corresponds to a send call, are undefined, with two exceptions: The error status field will contain valid information if the wait or test call returned with MPI_ERR_IN_STATUS; and the returned status can be queried by the call MPI_TEST_CANCELLED.* _Wait and _Test not working *does* imply, to me, that synchronous and asynchronous communication may not be mixed for the same communication. -tom

On Jun 12, 2009, at 1:16 PM, Nick Collier wrote:

I running into an issue where an irecv followed by a send results in deadlock. A simple test case,

class Item { private:

friend class boost::serialization::access;

template<class Archive> void serialize(Archive& ar, const unsigned int version) { ar & val; }

public: int val; Item() : val(1) { }

};

struct Receipt {

boost::mpi::request request; std::vector<Item> items; };

int main(int argc, char **argv) {

mpi::environment env(argc, argv); mpi::communicator world; Receipt receipt;

vector<Item> msg(100000);

int other = world.rank() == 0 ? 1 : 0; cout << world.rank() << " irecv from " << other << endl; receipt.request = world.irecv(other, 0, receipt.items); cout << world.rank() << " sending to " << other << endl; world.send(other, 0, msg);

receipt.request.wait();

cout << "Done" << endl; }

Run with mpirun -np 2, this never completes. It does complete with vector<Item> msg(10) however.

Nick

Looking at this issue the reason is probably be that for a general Item type Boost.MPI uses Boost.Serialization to send and receive serialized data. For that the receiving side has to resize a receive buffer after receiving the size of the serialized message. Boost.MPI currently first sends the size of that buffer and then the data in a second message. The irecv call only posts a receive for the first (size) message since it cannot receive the buffer yet. The receive for the buffer is called only in the request.wait() function, which we never get to because we are still stuck in the send call. This is an unfortunate design problem of Boost.MPI and there are two ways around it: 1) use the skeleton/content mechanism or send fixed-size arrays and an MPI datatype for Item 2) one could change irecv to use a single receive call - but then we need to give irecv an upper bound for the buffer needed to receive the serialized data, and the receive will fail if the size was too small. Would that behavior be preferred? Matthias