3

I am looking for an efficient one-sided communication library that can be used to implement RDMA efficiently.

Currently, I've looked at MPI-2 implementations such as MPICH2 and also MVAPICH2 (InfiniBand Architecture based implementation of MPI-2 that uses RDMA).

MVAPICH2 has a lot of functionality that I am looking for, unfortunately, I can't use that because I have no access to InfiniBand Architecture in my University currently.

So what I really want is, are there any other libraries (not necessary MPI implementations variants) that can be considered efficient in sense of reducing overhead between communicating processes (example: avoiding handshake in rendezvous protocol, thus offload overhead from the target process).

I would also appreciate any advice that helps me do something different instead of finding a cooked suitable library.

thanks.

razeh
  • 2,725
  • 1
  • 20
  • 27
LeTex
  • 1,452
  • 1
  • 14
  • 28

2 Answers2

5

First, RDMA and RMA are not the same thing. RDMA needs some sort of specialized hardware (such as InfiniBand) in order to access application buffers directly. If you do one-sided RMA operations via TCP for example, you are still going through the OS kernel and making several data copies in the process.

You could write your application using one-sided MPI operations, test it with an MPI library that runs on Ethernet, and when the time comes and you get access to faster hardware just replace the MPI library with something that supports RDMA.

Alternatively, look at some lower-level libraries such as ARMCI and GASNet. But nothing can do RDMA with no specialized hardware.

Is there a specific application you have in mind with these requirements?

Greg Inozemtsev
  • 4,516
  • 23
  • 26
  • 1
    Dear g.inozemtsev, thank you very much! I don't have any specific application requirement, I am just trying to address high-performance computations in a distributed environment where RDMA is involved (or as you have suggested efficient RMA is used). thanks for you suggestions. – LeTex Apr 26 '12 at 14:30
1

To supplement the answer of G.Inozemtsev, here are some more references.

Efficient one-sided communications over gigabit ethernet were investigated in several projects. There is iWARP (Wide Area RMDA Protocol over Internet) for which several GigE vendors provide hardware support.

There are several papers on iWARP from the group of D.K. Panda 1. P. Balaji et al. Supporting iWARP Compatibility and Features for Regular Network Adapters, 2005 2. S. Naravulla et al. High Performance MPI over iWARP: Early Experiences, 2007

Another option would be Berkeley Active Messages that can be implemented on top of UDP.

To my knowledge with the wide availability of InfiniBand fabrics this research direction is not actively developed.

Dima Chubarov
  • 16,199
  • 6
  • 40
  • 76