-1

Does rsync currently supports datta transfer over RDMA/Infiniband? I have to send some data to another server, but it's taking long to transfer, after searching for some time, i found there is somethng called as Infiniband network protocol which uses RDMA to send data and incurs low latency.

I'm wondering if it's possible to rsync over RDMA too? and if not, then How can we use Infiniband, does it has same API as rsync? It looks like Infiniband is restricted to some hardware based protocol internally.

ram
  • 83
  • 6

1 Answers1

2

You can configure the infiniband ipoib and use rsync over the ip connection. While this is not as fast as native infiniband it is faster than expected. I believe (could be wrong) that ipoib is limited to 10Gb/s thought that could be a limitation of my older cards.

For example, rsync over my infiniband ipoib connected between two systems seems to top out at 144MB/s Which is 1.15Gb/s. This is the limit of the drive write speed on the receiving drive.(a spinning HDD) I can get up to 500MB/s (4Gb/s) doing an SSD to SSD rsync over infiniband.

So at the moment, my limitations are the speeds at which I can read and/or write to the storage device. Not the speed of the infiniband connection. Meaning, connecting to a drive over my infiniband network is as fast as having the drive on the local machine.

Hope this helps some.

Joe Smith
  • 21
  • 2
  • 1
    Just wanted to quickly mention that the 10Gb/s limit you're seeing over IPoIB is probably either the Pcie bus limit (connectx-2, pcie 3.0, x4), or a single core CPU limit with newer cards, not the IPoIB itself. (I had to hunt for that one for awhile so figured I would share having seen your answer here when looking for something unrelated) – Joshua West Sep 09 '21 at 10:15