Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

InfiniBand(IB) is a switched fabric computer network communications link used in high-performance computing and enterprise data centers.

Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. InfiniBand host bus adapters and network switches are manufactured by Mellanox and Intel.

In addition to hardware manufactured by Mellanox and Intel there are a variety of software stacks available for the configuration and management of IB fabrics. Most major Linux distributions have some built-in support for Infiniband technologies, and the Open Fabrics Alliance(OFA) maintains the Open Fabrics Enterprise Distribution(OFED) which provides many software and firmware packages for the management, maintenance and setup of IB networks. In addition to the 'official' OFED maintained by the OFA, Mellanox and Intel both produce software stacks for their products that typically include their own customized versions of OFED.

112 questions
0
votes
0 answers

forwarding traffic between the ports

How to set up forwarding traffic between interfaces "HCA"? I want to implement a ring topology. PC Listing GUIDs: Port 0: 0x2c903000f6d27 Port 1: 0x2c903000f6d28 PC Listing GUIDs: Port 0: 0x2c90300048975 Port 1: 0x2c90300048976 PC Listing…
0
votes
1 answer

VMware infiniband configuration

I have a server which is intended to be a vmware host. This server has two Infiniband Mellanox ConnectX-2 dual-port cards. What I want to do is to make the following setup: The first card should work as a native Ethernet 10G network adapter, talking…
0
votes
0 answers

opensm CentOS 7 software switch

I have two computers both with infiniband cards in them that I would like to connect without a switch. I have a cable connecting the two computers through their QSFP ports. I have read the documentation and I see that opensm allows this. I have…
Ivan
  • 299
  • 1
  • 4
  • 13
0
votes
1 answer

Debian-based Infiniband point-to-point Voltaire dual port cards

I have the following configuration: I------------------I I------------------I I I(P0)---------(P0)I I I Voltaire HCA 0 I I Voltaire HCA 1 I I I(P1)---------(P1)I …
0
votes
0 answers

Getting eth_ipoib in Ubuntu

I've just discovered an ethernet driver called eth_ipoib which I'd like to experiment with. However, there appears to be no kernel module for it in the mainline. http://lwn.net/Articles/507258/ How do I get this going under Ubuntu server? I haven't…
hookenz
  • 14,472
  • 23
  • 88
  • 143
0
votes
0 answers

Possible to connect multiple InfiniBand switches?

Is it possible to connect multiple InfiniBand switches? If so, how can this be done? I've attempted to connect the switches directly via a cable, but that is not working, so I assume that I will need to configure the switches via the console, if…
cmorris14
  • 181
  • 2
  • 3
  • 11
0
votes
2 answers

Is ibdump in Mellanox OFED 2 supported in Ubuntu 14.04?

I have Ubuntu 14.04. I have Mellanox Technologies MT27500 Family [ConnectX-3] HCAs. I was able to pass all the tests like ibping and ib_rdma_lat. Now I'm running MPI tests to verify that InfiniBand is working. How can I actually see that there is…
Grigory
  • 167
  • 1
  • 2
  • 10
0
votes
1 answer

How much throughput increase to expect from link aggregation (LACP/802.3ad)?

I identified that data transfer within my cluster will be bottlenecked by the network interconnect (I'm saturating the dual SAS connection to my storage with sequential workloads), and I'm undecided between a 10 GbE or 40/56 Gb Infiniband to…
0
votes
1 answer

Cannot use SDP in Debian Squeezy

I have severe problems running SDP on Debian Squeeze. I'm using two machines with Mellanox adapters each. My /etc/modules looks like this: mlx4_ib # Mellanox ConnectX cards #ib_mthca # some mellanox cards #iw_cxgb3 # Chelsio T3 cards #iw_nes #…
Ulf
  • 387
  • 1
  • 5
  • 18
0
votes
4 answers

allow infiniband for non root users

I got Infiniband running on RHEL 6.3 [root@master ~]# ibv_devinfo hca_id: mthca0 transport: InfiniBand (0) fw_ver: 4.7.927 node_guid: 0017:08ff:ffd0:6f1c sys_image_guid: 0017:08ff:ffd0:6f1f vendor_id: …
user1219721
  • 487
  • 1
  • 6
  • 15
0
votes
3 answers

Areca 1880ix RAID hangs

Areca RAID controller ARC-1880ix-12 (firmware 1.50) hangs when on high load. My setup is: Chenbro 3U chassis Intel S5500BC mainboard Xeon 5603 CPU 16GB of RAM 12 Seagate SAS drives ST32000645SS (2 of them as hot spare, 10 as RAID10) Mellanox…
Dave
  • 51
  • 5
0
votes
1 answer

What is Infiniband?

What is Infiniband? What I have found is its a fabric switch network medium used for high performance computing. But more specificly what is it? Is it networking medium in the same way as cat5e is to ethernet? Or is it more like Fiber Channel where…
AtomicPorkchop
  • 1,975
  • 8
  • 34
  • 55
0
votes
1 answer

Ethernet connection is dropped when restarting openibd ( Infiniband ) service

I have a multiple servers with Ethernet controller on board and InfiniBand controller installed in a PCI slot. The problem is when im restarting openibd.service which should control only the infiniband adapter, for some reason, my ethernet network…
Oren
  • 21
  • 1
0
votes
0 answers

HPC node, Infiniband is DOWN

I have an HPC with 17 nodes running CentOS 7 and a dedicated Mellanox SX6036 Infiniband switch, each node has an Infiniband FDR interface. Recently one node started giving errors and a quick look showed that the ib0 IPoIB interface was down. 4: ib0:…
Chris Woelkers
  • 298
  • 2
  • 11
0
votes
0 answers

How can i set up my SunOracle Switch?

I am posting this question to see if anyone can help me. What I am trying to do is network my 2 hosts one being my proxmox machine and the other being my truenas scale machine into a sun oracle infiniband switch but I keep running into a dead end i…