MPI - Message Passing Interface

MPI - Message Passing Interface

What is MPI ?

MPI is a library used to send messages between processes in a distributed memory model.
These messages can contain all types of data such as string or integers but actual objects also.

It's not a programming language, it's a programming model that is mostly used for parallel programming in a cluster.

In this cluster, there is lot of nodes, but only one is the "chief", also known as master. Others are just workers nodes.

The primary advantage of using MPI is that it allow programmers to split a task and to distribute it to all or some specific workers nodes and allow each node to work on its own task simultaneously.

Okay ... Tell me more

The (crucial) Communicator

The Communicator is primordial to MPI projects, it allows to different processes to communicate with eachother. Of course, there can be several communicators in one MPI program, they will have theirs own processes. It can be useful if you want to split processes such that only certain processes receive messages to and from each other.

Actually, MPI allows two major types of communication :

• Point to Point

• Collectives

Point to Point

It's the most basic type of communication, which is when processes communicate on one to one basis. For example, P0 talks to P1 and P1 talks to P0.
Also, it can be blocking or unblocking. Obviously, blocking state means that the process will not continue until it has received its message. Otherwise, unblocking state means that it can keep executing steps while waiting for its message.

Point to Point work on two functions : Send and Receive.

• Send function can work on multiple types like : Primitives Types, Serializable Classes and Structures. But sending Serializable Classes can affect performance and sending will be slower than sending primitives type or structures.

• Receive function : When you receive a primitive type you must match it directly but not with classes.
  For example, you can receive a poodle as a dog.

Collective Communication

The other type of communication is Collective, all processes can communicate with each other.
It's a better type of communication than Point to Point because it would get messy pretty quickly.

There are two primary advantages of using Collective Communication :

• Code Readability and Maintainability :

• Its easier for programmers to read and maintain code

• Faster than Point to Point communications.
  For example, if you want to send a message to every processes it will use only one simple call, whereas it would require N^2 Point to Point communications, so it would consumes a lot of resources.

• Performance :

• Message Passing Interface has designed algorithms which are made for collective communication and like why just said before, it is very faster than Point to Point communication.

Majors Communication's Way with MPI

There are the majors communication's way that MPI implements :

• Barriers : "Wait for others before proceeding"

It just blocks all processes in a point, like a real barrier, and this barrier will open when all processes would be at that point.

• All-to-one : "All processes send messages to one"

It's when one process is asking all the other processes.

• One-to-all : "Sends messages to all processes from one"

It's when one process is sending a message to all the other processes.

• All-to-all : "All processes send messages to all processes"

It's when all processes are sending messages to each other.

• Combining results : "Get results from every process and do something with it"

It is used to combine data from all other processes and return it to the process who requested it.

MPI is used, for example, in Parallel Computing and Monte Carlo Simulations.