On Sun, Jan 18, 2009 at 8:46 PM, Mathias Gaunard <mathias.gaunard@ens-lyon.org> wrote:
Dean Michael Berris wrote:
I personally have dealt with two types of parallelization: parallelization at a high level (dealing with High Performance Computing using something like MPI for distributed message-passing parallel computation across machines) and parallelization at a low level (talking about SSE and auto-vectorization).
MPI is by no way high-level; it's low-level in that you have to explicitly say which tasks execute where and who they communicate with. Threads, for example, are much more high-level than that: they get scheduled dynamically, trying to use best the hardware as it is being used, or some other optimization factor, depending on the scheduler.
The difference between MPI and SIMD is not low-level vs high-level however: it's task-parallel vs data-parallel.
Yes, but the comparison I was making regarding high-level and low-level were in terms of code. In MPI you coded in C/C++ expressing communication between/among tasks without having to know about the actual (i.e. physical) topology of your architecture. This was considerably higher level than say writing hand-optimized SSE-aware code in your C++ code to express parallelism. I understand that MPI is a specification for communication primitives (much like assembly for distributed computing across logical/actual machines) and that wasn't the comparison I was going for when I meant it was "high-level". Though if you think about it, it's high level because you didn't deal directly with networking/shared-memory primitives, etc. but I don't want to belabor the point too much. ;-) When you talk about threads, those are high level in the context of single-machine parallelism -- unless your OS is a distributed OS that spawned processes across multiple machines and magically showed just one big environment to programs running in that parallel machine. Threads can be considered parallelism primitives in a single machine context, but not in a general parallel computing context where you might be spanning multiple distributed/independent machines. BTW, you can use asynchronous communication primitives in MPI and still use threads on each process. I don't even think MPI and Threads work at the same level anyway so it depends on which level you're looking from which determines whether you think MPI is low-level or Threads are high-level. ;-) -- Dean Michael C. Berris Software Engineer, Friendster, Inc.