1

I have never used RDMA. Though on single PCs, I have run commands that uses DMA for memory transfer. Can I use RDMA over ethernet ( I am using switch that support 10BASE-T, 100BASE-TX).

I will be using MPI to access data from two seperate machines independent of CPUs.

Is that possible? Or do I have to buy a special hardware?

gpuguy
  • 4,607
  • 17
  • 67
  • 125

4 Answers4

3

You'll need Ethernet cards that support RDMA, and you'll need switches that can work with these cards. And then you'll need MPI that can support protocols that the driver of those Ethernet cards provide.

The protocol (that I'm aware of, may be there are others) is RoCE, which is RDMA over Converged Ethernet (follow the links to get some details about this stuff).

MPI implementations that I know of that support RoCE are Open MPI (go here for FAQ), MVAPICH, and I'm sure there are others.

Have fun.

kliteyn
  • 1,917
  • 11
  • 24
1

I'm starting to support RDMA in Isis2 now (Isis2.codeplex.com), both on Windows and Linux. So that will be a second option for those who prefer not to use MPI. Of course I realize that you've posed this specifically about MPI and I mention it only because MPI is a bit inflexible (for example, not fault-tolerant).

My stuff offers a wide range of multicast and data replication options and is fault-tolerant. RDMA is just one of many configuration options.

Ken Birman
  • 1,088
  • 8
  • 25
0

RDMA over ethernet is called RoCE, It needs network interface card NIC, switches and cables. A company called Mellanox is one vendor of this kind of solution.

Mellanox also has a solution for RDMA high-bindwidth, low-latency network structure called InfiniBand.

You can find information on http://www.mellanox.com

And most MPI implementations support RDMA, including OpenMPI. I have used openmpi on RoCE, performance is much better than that of TCP/IP sockets.

And this is the official website for RDMA technologies: https://www.openfabrics.org

Brett Holman
  • 743
  • 6
  • 18
HY G
  • 185
  • 1
  • 14
0

If you have RDMA NIC, you can use RDMA in RoCE (RDMA over Converged Ethernet) mode. RoCE allows RDMA messages to be packaged in Ethernet frames.

But if you have only Ethernet NIC, you cannot use true version of RDMA. To be compatible with libraries that rely on RDMA verbs, you can consider Soft-RoCE. It provides a verbs API to user while sends data through Ethernet NIC, but with low performance compared with true RDMA.

James Ma
  • 1
  • 2