[Review Request] Message Passing Interface (MPI) Library
I am requesting a formal review for a Boost library that provides modern C++ bindings for the Message Passing Interface (MPI). The Message Passing Interface (MPI) is a standard interface for message passing in high-performance parallel applications. It defines a library interface, available from C, Fortran, and C++, for which there are many MPI implementations. Although there exist C++ bindings for MPI, they offer little functionality over the C bindings. The Boost.MPI library provides an alternative C++ interface to MPI that better supports modern C++ development styles, including complete support for user-defined data types and C++ Standard Library types and using arbitrary function objects for collective algorithms. Boost.MPI is built for efficiency, avoiding unnecessary overhead and translating high-level operations into the most specialized MPI call possible. Boost.MPI currently covers the majority of the MPI 1.1specification, including blocking and non-blocking point-to-point communication, collective operations, communicators, MPI datatypes, and environment management. The library documentation is available in HTML and PDF: http://www.osl.iu.edu/~dgregor/boost.mpi/libs/parallel/doc/html/ http://www.osl.iu.edu/~dgregor/boost.mpi/libs/parallel/doc/mpi.pdf You can browse the library source code here: http://www.osl.iu.edu/~dgregor/boost.mpi/ Or download the source code and documentation as a tarball here: http://www.osl.iu.edu/~dgregor/boost.mpi/boost-mpi-20060628.tgz The library is also available in the Boost Sandbox, in the subdirectories boost/parallel and libs/parallel. Note that you will need the latest Boost from CVS HEAD. Cheers, Doug Gregor
On 6/29/06, Doug Gregor <doug.gregor@gmail.com> wrote:
I am requesting a formal review for a Boost library that provides modern C++ bindings for the Message Passing Interface (MPI).
I've used MPI before, albeit in C, and I have a few questions (before even reading the documentation fully). I am also very excited with the prospect of actually having an MPI bindings implementation for C++ in boost. I see that this requires any one of three MPI implementations -- and there are a lot of differences between these implementations internally and architecturally. There's also the question of compiler compliance, and platform dependence -- although I haven't seen the code yet, my question would be more of pragmatic in the sense that it maintain a common STL-like interface without breaking the distributed/parallel computing frame. I would like to see really more of this -- though much like in parallel supercomputing applications, the issue really will be more of performance than anything. Anyone in the list can do a review right? -- Dean Michael C. Berris C/C++ Software Architect Orange and Bronze Software Labs http://3w-agility.blogspot.com/ http://cplusplus-soup.blogspot.com/ Mobile: +639287291459 Email: dean [at] orangeandbronze [dot] com
On Jun 28, 2006, at 9:29 PM, Dean Michael Berris wrote:
On 6/29/06, Doug Gregor <doug.gregor@gmail.com> wrote:
I am requesting a formal review for a Boost library that provides modern C++ bindings for the Message Passing Interface (MPI).
I've used MPI before, albeit in C, and I have a few questions (before even reading the documentation fully). I am also very excited with the prospect of actually having an MPI bindings implementation for C++ in boost.
Me too.
I see that this requires any one of three MPI implementations -- and there are a lot of differences between these implementations internally and architecturally. There's also the question of compiler compliance, and platform dependence -- although I haven't seen the code yet, my question would be more of pragmatic in the sense that it maintain a common STL-like interface without breaking the distributed/parallel computing frame.
Since the library is based on the published MPI 1.1 standard, there should be minimal portability issues provided you're using a conforming MPI implementation. As a quick check, I just tried it on our Sun-Fire cluster with Sun's HPC MPI and it worked okay. Just had to setup setenv MPICXX /opt/SUNWhpc/bin/mpCC setenv MPIRUN /opt/SUNWhpc/bin/mprun setenv MPIBUILD "<include>/opt/SUNWhpc/include <define>_MPIPP_H" The define is to skip inclusion of the MPI C++ headers. Thanks for the library Doug. I like the looks of your all_reduce, it looks very similar to what we did. -- Noel Belcourt
On Jun 29, 2006, at 12:47 AM, Noel Belcourt wrote:
As a quick check, I just tried it on our Sun-Fire cluster with Sun's HPC MPI and it worked okay. Just had to setup
setenv MPICXX /opt/SUNWhpc/bin/mpCC setenv MPIRUN /opt/SUNWhpc/bin/mprun setenv MPIBUILD "<include>/opt/SUNWhpc/include <define>_MPIPP_H"
Great, thanks for checking this! Doug
On Jun 29, 2006, at 7:53 AM, Douglas Gregor wrote:
On Jun 29, 2006, at 12:47 AM, Noel Belcourt wrote:
As a quick check, I just tried it on our Sun-Fire cluster with Sun's HPC MPI and it worked okay. Just had to setup
setenv MPICXX /opt/SUNWhpc/bin/mpCC setenv MPIRUN /opt/SUNWhpc/bin/mprun setenv MPIBUILD "<include>/opt/SUNWhpc/include <define>_MPIPP_H"
Great, thanks for checking this!
My pleasure. By the way, have you considered how your library might be used to propagate C++ exceptions thrown on one or more processors to the other processors? Robust parallel termination semantics are difficult to achieve and the ability to propagate C++ exceptions in parallel can greatly improve program behavior. Your library seems like a natural place to provide this capability. We've done some work in this area if you're interested. -- Noel Belcourt
On Jun 29, 2006, at 7:53 AM, Douglas Gregor wrote:
On Jun 29, 2006, at 12:47 AM, Noel Belcourt wrote:
As a quick check, I just tried it on our Sun-Fire cluster with Sun's HPC MPI and it worked okay. Just had to setup
setenv MPICXX /opt/SUNWhpc/bin/mpCC setenv MPIRUN /opt/SUNWhpc/bin/mprun setenv MPIBUILD "<include>/opt/SUNWhpc/include <define>_MPIPP_H"
Great, thanks for checking this!
My pleasure. By the way, have you considered how your library might be used to propagate C++ exceptions thrown on one or more processors to the other processors? Robust parallel termination semantics are difficult to achieve and the ability to propagate C++ exceptions in parallel can greatly improve program behavior. Your library seems like a natural place to provide this capability. We've done some work in this area if you're interested. -- Noel Belcourt
On Jun 29, 2006, at 11:49 AM, Noel Belcourt wrote:
By the way, have you considered how your library might be used to propagate C++ exceptions thrown on one or more processors to the other processors? Robust parallel termination semantics are difficult to achieve and the ability to propagate C++ exceptions in parallel can greatly improve program behavior. Your library seems like a natural place to provide this capability.
I've mused that such a thing might be possible, but I've never considered how it should be done or where it would fit into the library.
We've done some work in this area if you're interested.
Yes, I'd like to learn more about this. I don't expect we'd be able to support this feature in the initial version of the library (just covering MPI 1.1 requires quite a bit of effort, and we've had requests for MPI 2.0 features as well), but I think the (candidate) Boost.MPI would benefit from the ability to propagate C++ exceptions across processors. Doug
On Jun 29, 2006, at 5:29 AM, Dean Michael Berris wrote:
I would like to see really more of this -- though much like in parallel supercomputing applications, the issue really will be more of performance than anything.
The performance should be just that of the underlying MPI library and the hardware. The C++ bindings should have negligible overhead. The main performance-related design issue was about user-defined data types, for which an MPI_Datatype can be constructed using the Boost.Serialization library. The array serialization optimizations that have recently been added to the CVS HEAD of the Boost.Serialization library ensure that also this part can be done without performance penalties. This is also the reason why the latest Boost from the CVS HEAD is needed to compile and run the MPI library. Best regards Matthias
On Jun 28, 2006, at 11:29 PM, Dean Michael Berris wrote:
I see that this requires any one of three MPI implementations -- and there are a lot of differences between these implementations internally and architecturally.
In theory, the candidate Boost.MPI should work with any MPI that meets the MPI 1.1 specification. In practice, we tend to test with those three implementations.
There's also the question of compiler compliance, and platform dependence
You'll need a modern C++ compiler to use the candidate Boost.MPI. It should be platform-agnostic, but again--we only typically test a few platforms, namely x86 Linux, x86-64 Linux, and PowerPC Mac OS X.
-- although I haven't seen the code yet, my question would be more of pragmatic in the sense that it maintain a common STL-like interface without breaking the distributed/parallel computing frame.
I would like to see really more of this -- though much like in parallel supercomputing applications, the issue really will be more of performance than anything.
Although we have yet to run the tests with the candidate Boost.MPI, we ran NetPIPE numbers using a prototype of the same C++ interface. There was no impact on either bandwidth or latency.
Anyone in the list can do a review right?
I know there are a few Boosters that are in the HPC/parallel computing area. I'm also planning to strong-arm whatever MPI experts I can find :) Doug
On 6/29/06, Douglas Gregor <doug.gregor@gmail.com> wrote:
On Jun 28, 2006, at 11:29 PM, Dean Michael Berris wrote:
I see that this requires any one of three MPI implementations -- and there are a lot of differences between these implementations internally and architecturally.
In theory, the candidate Boost.MPI should work with any MPI that meets the MPI 1.1 specification. In practice, we tend to test with those three implementations.
API wise, I suppose this will be true. However, I'm really worried about the internal implementations of these MPI implementations and how some the the kinks will show from implementation to implementation. But then this concern is misplaced, and not really related to Boost.MPI, so I guess I don't need to go into too much detail on these concerns.
There's also the question of compiler compliance, and platform dependence
You'll need a modern C++ compiler to use the candidate Boost.MPI. It should be platform-agnostic, but again--we only typically test a few platforms, namely x86 Linux, x86-64 Linux, and PowerPC Mac OS X.
Looks like a pretty short list, but just right if you're dealing with Beowulf clusters. But then I don't really mind, because those are the only clusters I have experience with anyway. :)
-- although I haven't seen the code yet, my question would be more of pragmatic in the sense that it maintain a common STL-like interface without breaking the distributed/parallel computing frame.
I would like to see really more of this -- though much like in parallel supercomputing applications, the issue really will be more of performance than anything.
Although we have yet to run the tests with the candidate Boost.MPI, we ran NetPIPE numbers using a prototype of the same C++ interface. There was no impact on either bandwidth or latency.
I wouldn't really expect the candidate Boost.MPI bindings will make too much of a difference if it's going to be using the different existing MPI implementations. Like above, these concerns really are geared towards the actual MPI implementations instead of the candidate Boost.MPI Is there a consideration of using asio to implement the actual MPI standard, so that the candidate Boost.MPI will also implement the MPI standard in case there aren't any available MPI implementations for the platform? I'm thinking if asio will work on many different platforms (maybe even embedded devices), then maybe Boost.MPI can have a canned MPI implementation that comes along with it? Just a thought though, and I'd think this is too much a separate effort which might be a waste of time, but nonetheless might be a worthy goal.
Anyone in the list can do a review right?
I know there are a few Boosters that are in the HPC/parallel computing area. I'm also planning to strong-arm whatever MPI experts I can find :)
I'm no MPI expert, but have done some (read: very limited) HPC/parallel computing programming before. I'll love to see the discussion too, and be able to review it too when the process starts. I'm really excited with this, and hope I can be of help in any way I can be. Have a great day everyone! -- Dean Michael C. Berris C/C++ Software Architect Orange and Bronze Software Labs http://3w-agility.blogspot.com/ http://cplusplus-soup.blogspot.com/ Mobile: +639287291459 Email: dean [at] orangeandbronze [dot] com
On Jun 29, 2006, at 5:00 PM, Dean Michael Berris wrote:
Is there a consideration of using asio to implement the actual MPI standard, so that the candidate Boost.MPI will also implement the MPI standard in case there aren't any available MPI implementations for the platform? I'm thinking if asio will work on many different platforms (maybe even embedded devices), then maybe Boost.MPI can have a canned MPI implementation that comes along with it?
I, personally, am not at all interested in implementing the MPI standard. There are high-quality, high-performance, portable implementations available already (<shameless plug>Open MPI, for instance</shameless plug>). Building MPI on top of asio would require a *huge* investment, and we would not have a chance at beating existing MPI implementations in either portability or performance. No, it's better that we just provide the best C++ interface we can to the C library. That said, we can still try to *influence* MPI implementations, especially where there is a poor match between what C++ needs and what the C MPI bindings provide. For instance, MPI is very poor at receiving messages of unknown length. We could improve our implementation of the C++ bindings significantly if that issue could be addressed at a lower level. Doug
On 6/30/06, Doug Gregor <dgregor@cs.indiana.edu> wrote:
That said, we can still try to *influence* MPI implementations, especially where there is a poor match between what C++ needs and what the C MPI bindings provide. For instance, MPI is very poor at receiving messages of unknown length. We could improve our implementation of the C++ bindings significantly if that issue could be addressed at a lower level.
Sounds like a good thing to me. :) -- Dean Michael C. Berris C/C++ Software Architect Orange and Bronze Software Labs http://3w-agility.blogspot.com/ http://cplusplus-soup.blogspot.com/ Mobile: +639287291459 Email: dean [at] orangeandbronze [dot] com
Does this library make any use of the work done in OOMPI (see http://www.osl.iu.edu/research/oompi/) which also provides a C++ interface for MPI? John Doug Gregor wrote:
I am requesting a formal review for a Boost library that provides modern C++ bindings for the Message Passing Interface (MPI).
The Message Passing Interface (MPI) is a standard interface for message passing in high-performance parallel applications. It defines a library interface, available from C, Fortran, and C++, for which there are many MPI implementations. Although there exist C++ bindings for MPI, they offer little functionality over the C bindings. The Boost.MPI library provides an alternative C++ interface to MPI that better supports modern C++ development styles, including complete support for user-defined data types and C++ Standard Library types and using arbitrary function objects for collective algorithms. Boost.MPI is built for efficiency, avoiding unnecessary overhead and translating high-level operations into the most specialized MPI call possible. Boost.MPI currently covers the majority of the MPI 1.1specification, including blocking and non-blocking point-to-point communication, collective operations, communicators, MPI datatypes, and environment management.
The library documentation is available in HTML and PDF: http://www.osl.iu.edu/~dgregor/boost.mpi/libs/parallel/doc/html/ http://www.osl.iu.edu/~dgregor/boost.mpi/libs/parallel/doc/mpi.pdf
You can browse the library source code here: http://www.osl.iu.edu/~dgregor/boost.mpi/
Or download the source code and documentation as a tarball here: http://www.osl.iu.edu/~dgregor/boost.mpi/boost-mpi-20060628.tgz
The library is also available in the Boost Sandbox, in the subdirectories boost/parallel and libs/parallel.
Note that you will need the latest Boost from CVS HEAD.
Cheers, Doug Gregor _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Jun 29, 2006, at 10:29 AM, John Fletcher wrote:
Does this library make any use of the work done in OOMPI (see http://www.osl.iu.edu/research/oompi/) which also provides a C++ interface for MPI?
I work in the Open Systems Lab, which is also responsible for OOMPI, so we're well-aware of OOPMPI and there's definitely some institutional mindshare. That said, OOMPI and the candidate Boost.MPI take rather different approaches to presenting MPI in C++. Architecturally, OOMPI is, well, OO: it relies on inheritance, virtual functions, and overloading for abstraction. Boost.MPI is build using Generic Programming, so it relies on templates, traits, and specialization for abstraction. Boost.MPI is meant to fit in best with the style of programming used in STL and Boost, with function objects, integration with Boost.Serialization, etc. I think there's also a philosophical difference between the two. OOMPI provides some syntactic innovations (such as ports) that can make MPI easier to use. Boost.MPI is a little more "purist", attempting to keep the flavor of the C MPI bindings while adapting better to modern C++. We still "send" in the same way as "MPI_Send" in Boost.MPI. Of course, should something like ports be useful, they could be added as syntactic sugar. Doug
Doug, Thank you that is very helpful. John Doug Gregor wrote:
On Jun 29, 2006, at 10:29 AM, John Fletcher wrote:
Does this library make any use of the work done in OOMPI (see http://www.osl.iu.edu/research/oompi/) which also provides a C++ interface for MPI?
I work in the Open Systems Lab, which is also responsible for OOMPI, so we're well-aware of OOPMPI and there's definitely some institutional mindshare. That said, OOMPI and the candidate Boost.MPI take rather different approaches to presenting MPI in C++. Architecturally, OOMPI is, well, OO: it relies on inheritance, virtual functions, and overloading for abstraction. Boost.MPI is build using Generic Programming, so it relies on templates, traits, and specialization for abstraction. Boost.MPI is meant to fit in best with the style of programming used in STL and Boost, with function objects, integration with Boost.Serialization, etc.
I think there's also a philosophical difference between the two. OOMPI provides some syntactic innovations (such as ports) that can make MPI easier to use. Boost.MPI is a little more "purist", attempting to keep the flavor of the C MPI bindings while adapting better to modern C++. We still "send" in the same way as "MPI_Send" in Boost.MPI. Of course, should something like ports be useful, they could be added as syntactic sugar.
Doug _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Doug Gregor wrote:
I think there's also a philosophical difference between the two. OOMPI provides some syntactic innovations (such as ports) that can make MPI easier to use. Boost.MPI is a little more "purist", attempting to keep the flavor of the C MPI bindings while adapting better to modern C++. We still "send" in the same way as "MPI_Send" in Boost.MPI. Of course, should something like ports be useful, they could be added as syntactic sugar.
I'm not terribly familiar with OOMPI but my brief look-see got me interested in the ports feature. Something like this on top of Boost.MPI would be great. Even better would be a communication interface that could work with various underlying technologies (MPI, Boost.asio, Boost.interprocess, etc.). I don't know how practically realizable it is to do with while maintaining performance, though. A while back I did some experiments with a colleague to hack up a UPC compiler to recognize UPC constructs in C++. It turns out that one can do a bunch of nifty things with template metaprogramming in the context of parallel computation in the style of UPC. I wonder if it would be possible to emulate something like a UPC++ with creative operator overloading, a parallel communication library and template metaprogramming wizardry? I don't think it would be possible to approach the speed of a native UPC++ but it might be interesting for some machine architectures such as clusters or other low-bandwidth systems. -Dave
Hi Doug, I have received your request and will add the library to the review queue. Cheers, ron On Jun 28, 2006, at 11:01 PM, Doug Gregor wrote:
I am requesting a formal review for a Boost library that provides modern C++ bindings for the Message Passing Interface (MPI).
Does Boost.MPI support MPICH2 ? Regards, Philipp On 6/30/06, Ronald Garcia <garcia@cs.indiana.edu> wrote:
Hi Doug,
I have received your request and will add the library to the review queue.
Cheers, ron
On Jun 28, 2006, at 11:01 PM, Doug Gregor wrote:
I am requesting a formal review for a Boost library that provides modern C++ bindings for the Message Passing Interface (MPI).
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Hi Doug, I'm just thinking aloud here: This might not be directly related, but will the candidate Boost MPI implementation implement the common parallel architectural patterns (like meshes, lattices, trees, rings, and manager-worker/master-slave, among others), using generative techniques (or even template metaprogramming) to perhaps make the common solutions readily usable "out of the box" ? It might be a good value add over the other C++ wrappers to MPI implementations, but I guess at this point getting the libarary into Boost is more a focus than features/value-add. Will the above be considered in the future perhaps, as part of the candidate Boost.MPI library? Or will that be beyond the scope of the library? -- Dean Michael C. Berris C/C++ Software Architect Orange and Bronze Software Labs http://3w-agility.blogspot.com/ http://cplusplus-soup.blogspot.com/ Mobile: +639287291459 Email: dean [at] orangeandbronze [dot] com
Hi Dean, On Jul 3, 2006, at 3:42 PM, Dean Michael Berris wrote:
Hi Doug,
I'm just thinking aloud here:
This might not be directly related, but will the candidate Boost MPI implementation implement the common parallel architectural patterns (like meshes, lattices, trees, rings, and manager-worker/master-slave, among others), using generative techniques (or even template metaprogramming) to perhaps make the common solutions readily usable "out of the box" ?
It might be a good value add over the other C++ wrappers to MPI implementations, but I guess at this point getting the libarary into Boost is more a focus than features/value-add. Will the above be considered in the future perhaps, as part of the candidate Boost.MPI library? Or will that be beyond the scope of the library?
I think implementations of common parallel architectural patterns would make an excellent library on top of Boost.MPI (or asio, or threads, or all 3), but I'd say it's out of scope for Boost.MPI itself. Doug
Doug Gregor wrote:
I think implementations of common parallel architectural patterns would make an excellent library on top of Boost.MPI (or asio, or threads, or all 3), but I'd say it's out of scope for Boost.MPI itself.
I agree with this. I'm interested in such things and it would be great to get a library that can operate with a variety of communication mechanisms (MPI, UPC(++?), etc.). -Dave
On 7/3/06, David Greene <greened@obbligato.org> wrote:
Doug Gregor wrote:
I think implementations of common parallel architectural patterns would make an excellent library on top of Boost.MPI (or asio, or threads, or all 3), but I'd say it's out of scope for Boost.MPI itself.
I agree with this. I'm interested in such things and it would be great to get a library that can operate with a variety of communication mechanisms (MPI, UPC(++?), etc.).
Shouldnt this high-level library be incorporated by interprocess?
-Dave
-- Felipe Magno de Almeida
On 7/4/06, David Greene <greened@obbligato.org> wrote:
Doug Gregor wrote:
I think implementations of common parallel architectural patterns would make an excellent library on top of Boost.MPI (or asio, or threads, or all 3), but I'd say it's out of scope for Boost.MPI itself.
I agree with this. I'm interested in such things and it would be great to get a library that can operate with a variety of communication mechanisms (MPI, UPC(++?), etc.).
Now the only thing missing really is a cluster on which to test the concepts on to... :D -- Dean Michael C. Berris C/C++ Software Architect Orange and Bronze Software Labs http://3w-agility.blogspot.com/ http://cplusplus-soup.blogspot.com/ Mobile: +639287291459 Email: dean [at] orangeandbronze [dot] com
participants (11)
-
David Greene -
Dean Michael Berris -
Doug Gregor -
Doug Gregor -
Douglas Gregor
-
Felipe Magno de Almeida -
John Fletcher -
Matthias Troyer -
Noel Belcourt -
Philipp Robbel -
Ronald Garcia