there are many papers mentioning that the RDMA SEND/RECV is often used for command/signal or small-size data communication. But I can't find a reasonable explanation for that.
I know that the memory semantic (WRITE/READ) can reduce latency and CPU workload further (compared with SEND/RECV), and it seems that the WRITE/READ operations can simply dominate in all cases?
RDMA SEND/RECV are used for two sided communication while RDMA WRITE/READ is used for one sided communication. Hence the performance advantage with WRITE/READ but the usage makes it little bit more complex (or maybe the usecase demands to use SEND/RECV) than SEND/RECV.
For example, In RDMA read, during communication, the sender remains passive while the receiver posts the RDMA Read request. The challenge is to ensure that the passive side (the sender) does not write memory that is currently being read by the active side (the receiver).
Look here for some detailed explanations.
