Hi Boosters!
In this post I just wanted to throw in my two cents about the newly
accepted MPI library, from a luser's point of view.
(doc: http://tinyurl.com/fjc9x)
It was a great excitement to see the news on boost.org "Message Passing
Review Begins"! The C bindings of MPI are a great pain to use -- a fact
which is mostly denied by programmers who are unaware of C++'s beauty
and content with even good old F77 most of the time. And, as it has been
pointed out by Boost.MPI developers, the C++ bindings of standard MPI,
though an improvement, are nothing more than simple wrappers on top of
the C bindings. Moreover, certain supercomputing centers do not even
have these C++ bindings of MPI installed on their otherwise
state-of-the-art clusters.
So, anyway, it was great to hear that finally MPI will be *boost*ed.
There is a lot to what is good about this new boost library. For one,
quoting Jeremy: "One particularly nice aspect of Boost.MPI is the way it
leverages the Boost serialization library". This indeed appears to be a
greatly elegant way to message-pass user types around. There are other
features to applaud, such as reduce() via arbitrary functors. Yet, in
this post, with the intent of providing constructive criticism, I'd like
to point out several design issues related to the user interface; mostly
in comparison with other MPI implementations.
(hereafter the discussion will assume specific knowledge about MPI)
Here is a brief outline of what I'll be talking about, somewhat in the
order of importance:
1. Port object
2. Message object
3. Skeleton and content questions
4. MPI-2 bindings
*Port Object*
Those of you who have actively programmed with MPI certainly have used
or at least heard of the OOMPI library (I believe Doug Gregor, one of
the two developers of Boost.MPI, is affiliated with the institution
OOMPI was born). Its web page, http://www.osl.iu.edu/research/oompi/ ,
greets you with random (mostly funny) messages, such as "OOMPI- Isn't it
time you did something for YOU?" :)
Anyway, it was thanks to OOMPI library that I realized that in fact MPI
programming could be fun! Especially, there is this *OOMPI_Port*
abstraction that literally _enlighten_s the way point-to-point (PTP) and
rooted-collective (RC) operations are made. Let's contrast the way PTP
is carried out for the three interfaces (Standard MPI, Boost.MPI and OOMPI):
//================================================\\
// Standard way
int rank;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
int msg;
int msg_tag = 10;
if (rank == 0) {
msg = 17;
MPI_Send(&msg, 1, MPI_INT, 1, msg_tag, MPI_COMM_WORLD);
// (synopsis: http://tinyurl.com/ejmgg )
} elseif (rank==1) {
MPI_Recv(&msg, 1, MPI_INT, 0, msg_tag, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
// (synopsis: http://tinyurl.com/elkez )
}
// Boost.MPI way
mpi::communicator world;
int msg;
int msg_tag = 10;
if (world.rank() == 0) {
msg = 17;
world.send(1, msg_tag, msg);
} elseif (world.rank() == 1) {
world.recv(0, msg_tag, msg);
}
// OOMPI way
OOMPI_Comm_world& world = OOMPI_COMM_WORLD;
world.Init(argc,argv);
int msg;
int msg_tag = 10;
OOMPI_Port to (world[1]);
OOMPI_Port from (world[0]);
if (world.Rank() == from.Rank()) {
msg = 17;
to.Send(msg,msg_tag);
} else if (world.Rank() == to.Rank()) {
from.Recv(msg,msg_tag);
}
//=============================================
The standard C provides global functions for this simplest and most
commonly employed Send/Recv commands. These functions require a long
list of arguments, which are usually a pain to keep track of (here is
another P2P command which is really harsh http://tinyurl.com/mk6dq).
Also, such an interface does not provide any conceptual description of
what is going on. The design of MPI is indeed mostly object oriented
(http://tinyurl.com/qm9hd), however the implementation doesn't realize
this fact.
Boost.MPI improves two aspects of this: For one the P2P commands are
member functions of the communicator (same as the standard C++
bindings); and two, the message's type is templated and the user does
not have to repeatedly specify what she is sending. But, as in
C-version, one still has to realize the points (P's in P2P) of
communication as a bunch of integer values rather than, well, the points
of communication, which are an entities by themselves.
OOMPI takes this one step further and things kind of fit in place with
the whole framework of MPI's design: The points of communication are
objects themselves, abstracted as OOMPI_Port class. P2P functions are
member functions of OOMPI_Port and OOMPI_Port is in turn a certain piece
of a given communicator. Here is how an RC communication looks like:
//=============================================
// Root-collective communication with OOMPI.
// The message will be created at the 4th process and
//+ will be broadcasted to everyone in the world communicator.
OOMPI_Port root = world[4];
root.Bcast(msg);
// or if the Port will not be used for anything else, it is
//+ possible to create a temporary:
world[4].Bcast(msg);
// Here is how this is with Boost.MPI
broadcast(world, msg, 4);
//=============================================
Clearly the OOMPI version is more instructive of the communication
process. A more elaborate illustration can be given in the context of
communication topologies (a feature that is not yet available in
Boost.MPI, unless I am mistaken). For instance:
//=============================================
// Create a cartesian virtual topology
enum {X,Y,NDIM};
int shape[NDIM] = {3,2}; // requires world.size() >= 6;
bool is_periodic[] = {true,true};
OOMPI_Cart_comm grid(world,NDIM,shape,is_periodic);
// Get nodes of subsequent P2P communication
OOMPI_Port left = grid[grid.Shift(X,-1)]; // left neighbor
OOMPI_Port right = grid[grid.Shift(X, 1)]; // right neighbor
// get cartesian coordinates
int mycoords[NDIM]; grid.Coords(mycoords);
// create a message
int msg = grid.Rank();
// P2P communicate
left.Send(msg); // send to left
right.Recv(msg); // recv from right
//================================================
I wonder why this Port abstraction (or a similar idea) couldn't make its
way to Boost.MPI? Is there an underlying design decision to this?
Also, why the P2P commands are members of the communicator class but RC
commands such as broadcast and gather are global?
*Message object*
A message in MPI is created by specifiying a data type, data count, data
address and finally a message tag. Note the C interface of passing a
message in a P2P communication (see first example above). The 4 out of 6
arguments in MPI_Send() is just to specify the message. And for every
communication statement of the same kind, this list should be repeated.
Boost.MPI (ignoring the skeleton/content idea for a second) reduces this
to three arguments (http://tinyurl.com/k2w7q): Reference to the data
(from which the type is deduced), data count and the message tag.
However, this message is still not a separate entity; and all this
information is repeatedly specified as arguments to communication functions.
OOMPI provides the convenience of OOMPI_Message object, which is then
used in communications. For instance, to pass an array of integers:
//=============================================
int size = 5;
int arr[size];
OOMPI_Tag msg_tag = 10;
OOMPI_Message msg(arr1,size,msg_tag);
if (world.Rank() == root.Rank()) /* fill up the arr */;
root.Bcast(msg);
//=============================================
*Skeleton and Content - Separating structure from content*
Boost.MPI provides this promising functionality which not only
simplifies message specifications during communication calls but also
helps to easily create more complex data types.
But as admittedly a naive user, I am not so sure if this won't actually
degrade the performance, or why is this even necessary at all. For
instance, let's examine the example given here
(http://tinyurl.com/mgwkt). A plain C implementation of the same thing
would be something like this:
//===================================================
int root = 0;
// Create the message type (this section of the
//+ code is run by ALL processes in the communicator).
int list_len = 10; // size of the message
std::list<int> mylist(list_len);
std::vector