The purpose of Remote Memory Access (RMA)

Question

I read MPI standard and found a section with MPI_Put and MPI_Get operations. However, it is not clear for me what benefits gives RMA that lays in the foundation of those functions. When it is better to use such functions instead of asynchronous MPI_Isend, MPI_Irecv? What is a general purpose of use RMA (not in MPI context)?

Answer 1

The largest perk of One-Sided communications is the ability to push or fetch data between nodes without the overhead of send/receive.

Imagine a scenario with 2 nodes. You want to push data from node 1 to node 2.

When node 1 sends data to node 2 using MPI_ISend or MPI_Send, node 2 must still formally receive that data using MPI_Irecv. Node 2 is going to wait on that receive, match the MPI tags, and then allow you to continue. This is similar in theory to sockets in that both sides have to coordinate. Even with async versions MPI_I*, there is still coordination.

With MPI_Put, you push the data from node 1 and node 2 will have it. There are synchronization steps involved, but the punchline is that node 2 will have it and not need to formally accept. All nodes in the group have access to the memory buffers attached by the other nodes.

A colleague of mine had a good analogy. MPI_Send and MPI_Recv are like mail delivery where the recipient has to sign for the package. They can wait at the door indefinitely (MPI_Send) or leave a note for you to call back (MPI_ISend).

MPI_Put is like the Amazon delivery person walking into your house and putting the groceries away. MPI_Get is like you walking into a grocery store when it is closed and just grabbing what you need.

Similar to this example, MPI_Put and MPI_Get rely on trust and good design. They should only be used when shared memory and one-direction data movement are effective.

Good Overview: https://www.cc.gatech.edu/~echow/ipcc/hpc-course/19_rma.pdf

Answer 2

Some applications are easier or more natural to write using RMA. For example, consider a 2D simulation application where each process owns some part of a grid but requires data from other processes in order to compute its iteration. Using message passing primitives requires each process to match send/recv pairs and each receiver to understand the meaning of each message (please write this data here). The programmer has to carefully avoid deadlocks using coordinated sends and receives, or what is often called "asynchronous" Immediate operations. Using RMA primitives doesn't require the target message to understand the request as its meaning is implied (please give me or write this data).

However, it is not correct to see the MPI_Put and MPI_Get operations as truly one-sided. MPI was designed to work on different hardware. Some networks such as infiniband support true DRMA in the sense that they don't require involvement from the target process (and as I read here, it is not truly the case). Other networks don't support RDMA and the RMA standard was designed so that efficient RMA primitives (relatively to hardware capabilities) could be implemented without contradicting this specification.

As explained in the Using advanced MPI book, MPI_Put is not just a send that doesn't require a recv, but it is similar to a MPI_Isend that requires a call such as MPI_Wait or MPI_Test to make progress even if a corresponding MPI_Recv variant was posted. Similarly, MPI RMA operations require synchronization operations such as MPI_Win_fence for the most basic one where progress can be made. The MPI standard only says that these operations should be completed at its latest when one such operation is called, implying it could be completed before. The behavior depends on the MPI implementation and RMA operations may actually be implemented using message passing.