1

I have difficulties to understand how and in which cases RDMA operations are used.

Let's say we have a server and a client. The client writes data via rdma-write to the memory region of the server. Since the server doesn't get any notification that data arrived during the client-side rdma-write operation (without immediate), I wonder now: How can the server access this data if it doesn't even know that it got some, let alone where it is located (in the memory region)?

In my research I only found examples and explanations simply describing how to send/read data via rdma-write/read, but no one explained e.g. how to use the data accordingly.

1 Answers1

1

The server's CPU needs to be notified in a separate message abuot the data's arrival before accessing it, either using a subsequent RDMA write with immediate operation, a send operation, or an atomic operation.

haggai_e
  • 4,689
  • 1
  • 24
  • 37
  • Assuming you would do a rdma-write-with-immediate operation from the beginning, how does the performance look like if you compare it to the send/receive operation? In both cases a receive request is needed on the receiver side which is processed either by the immediate or the send/receive operation. So I would assume that the performance here is not very different right? Are there any publications worth mentioning on this topic? –  Mar 14 '21 at 15:54
  • Yes, I don't think performance would be different. The difference is that RDMA write allows direct placement of the data, so it facilities zero-copy transfers. You could also poll on memory written by an RDMA write, without the immediate, to get better performance, but this is not supported by the spec, so whether it works or not is implementation dependent. – haggai_e Mar 15 '21 at 06:55
  • There are many papers about the tradeoffs of RDMA operations. I suggest starting with "Design Guidelines for High Performance RDMA Systems" by Kalia et al., "Storm: a fast transactional dataplane for remote data structures" by Novakovic et al., and "RDMA Read Based Rendezvous Protocol for MPI over InfiniBand: Design Alternatives and Benefits" by Sur et al.. – haggai_e Mar 15 '21 at 07:02