0

I am planning to connect 18 nodes together with infiniband what do I need 1 36 port switch ? or do I need 2 switches ? 1 HCA for each node ? dual port ? 1 cable for each node ? or do I need 2 can the switch handle the subnet management or will it have to be on a host. if it can be on the switch is this recommended ? how do I go about connecting this infiniband network to the existing Ethernet network? can I leave the existing Ethernet connections to the hosts ?

user945571
  • 1
  • 1

1 Answers1

4

for most of your questions you have multiple options and in many cases it depends on your requirements: here are some input:

  1. it depends what kind of fault tolerance you like to have: if you are happy with one HCA with dual ports then go for it, otherwise use 2 HCAs: But automatic path fail over (APM as defined in IB spec) didn't work across HCAs during initial days(spec needed change for that), so if you like to have automatic path fail over then look for some higher level solution: this feature is very cool.

Now your cable question is related to this: of course you need one cable per port to connect to switch: so in your case, you may need 1 to 4 cables per host depending on how many ports you plan to use per host.

  1. swtich question: if you use one port per switch then there are 18 port switches from popular vendors, if you need a bit more reliability then 2 x 18 port switches make sense.

  2. I end up running SM on hosts for different issues. running SM on host is fine, I have used it in both ways.

  3. There are InfiniBand switches with ethernet gateway built-in to bridge traffic between ethernet and InfiniBand: you will use one of those:: protocols can be ipoib and/or SDP: SDP (Socket direct protocol) gives you good RDMA capabilities and performance will be better than ipoib.

  4. I don't want to name any vendors here, but IBTA website (link given below) or a simple google search will give you those pointers... http://www.infinibandta.org/

hope this helps, cforfun

cforfun
  • 135
  • 1
  • 11