Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

InfiniBand(IB) is a switched fabric computer network communications link used in high-performance computing and enterprise data centers.

Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. InfiniBand host bus adapters and network switches are manufactured by Mellanox and Intel.

In addition to hardware manufactured by Mellanox and Intel there are a variety of software stacks available for the configuration and management of IB fabrics. Most major Linux distributions have some built-in support for Infiniband technologies, and the Open Fabrics Alliance(OFA) maintains the Open Fabrics Enterprise Distribution(OFED) which provides many software and firmware packages for the management, maintenance and setup of IB networks. In addition to the 'official' OFED maintained by the OFA, Mellanox and Intel both produce software stacks for their products that typically include their own customized versions of OFED.

112 questions
2
votes
0 answers

CentOS 7 - Systemctl dependency problems with network mount and OpenSM

I have the follwoing problem. I want to mount a glusterfs share over Infiniband on an CentOS 7 client. The glusterfs share '/data' is severed by a host called storage1. For the client I I setup and minimal CentOS system. I installed rdma and opensm…
M.K. aka Grisu
  • 141
  • 1
  • 8
2
votes
3 answers

Is it possible to get any kind of unique identifier for the Infiniband switch that a particular Linux box is connected to?

I am trying to identify which Infiniband NIC's are connected to which switches, in a manner that is very similar to this question. However, the solution proposed there is not working for Mellanox Infiniband switches, and I have not been able to find…
merlin2011
  • 273
  • 1
  • 12
2
votes
0 answers

Can two infiniband ports be bridged together to make a switch?

I have 3 machines with infiniband cards, two with single ports and one with two ports. I do not own any switches. I have my network setup such that the computer with 2 ports, running Linux, routes between the other two hosts in different IP subnets.…
Dougvj
  • 121
  • 3
2
votes
0 answers

Infiniband link is at 4X SDR with DDR HCAs; has anyone seen this before?

I've recently upgraded my Infiniband HCAs from SDR to DDR, however upon upgrading, it has not improved the link speed at all. Receiving the following output from 'iblinkinfo': CA: ip-san-1-au3a mthca0: 0x0008f104039812b5 3 1[ ] ==( 4X …
jaseeey
  • 1,462
  • 16
  • 20
2
votes
1 answer

How to configure hosts to show host name in ibnetdiscover (infiniband)?

Windows machines with the OFED stack installed show the host names in ibnetdiscover. Under Ubuntu Linux all I see is "MT25408 ConnectX Mellanox Technologies". Is there a way of configuring the driver to show the hostname or some other text…
hookenz
  • 14,472
  • 23
  • 88
  • 143
2
votes
2 answers

Infiniband, SRP and failover

I am considering an linux-based infiniband storage setup, but I am tripping over whether or not I can do active/passive or active/active failover with DRBD and SRP targets. I am led to believe that SRP is the real problem here and that it may be…
nrb
  • 93
  • 1
  • 6
2
votes
0 answers

How do I make dnsmasq serve IP addresses via IPoIB?

I have a cluster farm that I'm setting up. The nodes (computers in the farm) are connected via ethernet & IP over Infiniband. I'm needing to netboot the nodes and thought dnsmasq would fit well as it provides all the features including support for…
hookenz
  • 14,472
  • 23
  • 88
  • 143
2
votes
2 answers

How to correctly set up iWARP? Preferably on loopback

iWARP is a protocol for doing remote direct memory access (RDMA) on top of TCP/IP, so that it can work with Ethernet and other network types as opposed to Infiniband. It works with many of the standard IB interfaces - the IB verbs, for example - so…
ajdecon
  • 1,301
  • 4
  • 14
  • 21
2
votes
2 answers

How to set up an Infiniband SRP Target?

I want to set up a server as a SCSI Target. How good are the open source implementations? What is the tutorial one? How is performance and reliability? How dependent is the OS? How much is built in typically? I was looking at CentOS How…
eruditass
  • 159
  • 1
  • 8
2
votes
2 answers

Infiniband UIO vs Infiniband HCA

What is the fundamental difference between: Infiniband Universal I/O Card (e.g. Supermicro AOC-UINF-M2) and Infiniband Host Channel Adapter (e.g. Qlogic QLE7240-CK) Can't both of those do do IP-over-IB?
Aleksandr Levchuk
  • 2,465
  • 3
  • 22
  • 41
2
votes
0 answers

highly available storage over infiniband: what other than mdraid?

is mdadm over infiniband a bad idea? what is the real trick to get reasonable performing storage to survive a single machine failure? We have been running ceph for a few years now and its great for easy (ish) redundancy, but its performance is eye…
aep
  • 177
  • 2
  • 10
2
votes
0 answers

Infiniband fabric with 3 nodes - newbie

I am trying to connect 3 HP z840 workstations using: Mellanox ConnectX-3 VPI 40 / 56GbE Dual-Port QSFP Adapter MCX354A-FCBT Mellanox SX6005 12-port Non-blocking Unmanaged 56Gb/s Description of machines to be connected: oak-rd0-linux (main node…
theenemy
  • 121
  • 2
2
votes
1 answer

Unable to get Mellanox ConnectX-3 to work with transceiver/cable

I'm attempting to upgrade my local network setup to 40G but I seem to have hit a snag. I have Mellanox ConnectX-3 VPI CX353A network cards I got off ebay, updated their firmware and installed their drivers. After switching transceivers/cable and…
2
votes
1 answer

How to access files from a local server with infiniband without using IP

I want to start of by saying that I'm very inexperienced with everything I'm doing so please take it easy on me. I have 4 computers each with Red hat Enterprise Linux server 7.8 installed, they are all connected to a Voltaire grid director 4036…
tot18
  • 23
  • 3
2
votes
2 answers

Infiniband drivers : OFED or distro included?

I'm setting up a Linux cluster with infiniband network, and I'm quite a newby in infiniband wolrd, any advice is more than welcome ! We are currently using Mellanox OFED drivers, but our infiniband cards are old and not recognized by the latest…
nirnaeth
  • 33
  • 6