2

I want to start of by saying that I'm very inexperienced with everything I'm doing so please take it easy on me.

I have 4 computers each with Red hat Enterprise Linux server 7.8 installed, they are all connected to a Voltaire grid director 4036 infiniband switch thats running the subnet manager. For simplicity's sake I will call the computers s1-s4. I want s2, s3 and s4 to be able to access and download files from s1 without using IP since I've read that removing the use of IP's can improve performance by up to 20%(correct me on this if I'm wrong). Is it even possible to make file transfers without the use of IP?

Also, since I'm new to this site feel free to tell me if I should have given more information or was unclear about something. If you think I have misunderstood something then you can also feel free to correct me on it.

Thanks

tot18
  • 23
  • 3
  • 1
    Why? Why do you wish not to use an infiniband-specific protocol over an infiniband network in favor of a different protocol? Explaining the why can help us find the best solution for you. – music2myear Mar 31 '21 at 22:00
  • @music2myear Yes, I edited the post to make it much clearer as to why I didnt want to use IPoIB. I've read that it can improve performance by a decent amount when not using IP addresses. –  Apr 01 '21 at 17:06
  • "I want to start of by saying that I'm very inexperienced with everything I'm doing so please take it easy on me." - as per site rules this is a place for professionals and we do not provide links or references to basic documentation. This sounds like a very basic question and is not unsimilar to "how do I set up an IP4 network" - which would be off topic per site rules. – TomTom Apr 04 '21 at 22:08

1 Answers1

1

tl;dr: look and stick with NFSoRDMA.

That's some points that needs to be clear. Considering that you're using RHEL 7 as basis you may want to serve files over NFS from s1 to s2, s3 and s4.

Your right about running IP over Infiniband is problem due to performance and latency issues, not to mention the high CPU load, since basically there's no IP offload on Infiniband, everything is done in software. When running pure IPoIB workload you'll probably have a server with 99% wasted CPU cycles on ksoftirq.

With that in mind, there's another issue: NFS, and also other file services, relies on IP networks to properly function.

What you want is an RDMA enabled protocol for file sharing. Usually they still depends on IP, but only for initial handshake, after the connection is stablished, they got upgraded to RDMA, and in your case, using Infiniband. There's also other fabrics that implements RDMA: RoCE and Omni-Path for instance.

For NFS specifically you should look for NFSoRDMA (NFS over RDMA). You didn't say which hardware do you have, but if it's Mellanox you'll need to install MLNX OFED (probably the LTS 4.9 version to support old hardware) and enable NFSoRDMA. I'm not sure if NFSoRDMA is available on the INBOX RHEL 7 OFED implementation. On EL8 is available AFAIK.

You'll still need an up and running IPoIB interface with an IP address, usually ib0, but with proper configuration NFS will only transfer control messages over IP while the bulk traffic will be over RDMA.

Other filesystems, for example, BeeGFS operates the same way. The initialize over IP and upgrades to RDMA. Gluster was an option in the past but RDMA has been removed from it.

Vinícius Ferrão
  • 5,520
  • 11
  • 55
  • 95