Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

InfiniBand(IB) is a switched fabric computer network communications link used in high-performance computing and enterprise data centers.

Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. InfiniBand host bus adapters and network switches are manufactured by Mellanox and Intel.

In addition to hardware manufactured by Mellanox and Intel there are a variety of software stacks available for the configuration and management of IB fabrics. Most major Linux distributions have some built-in support for Infiniband technologies, and the Open Fabrics Alliance(OFA) maintains the Open Fabrics Enterprise Distribution(OFED) which provides many software and firmware packages for the management, maintenance and setup of IB networks. In addition to the 'official' OFED maintained by the OFA, Mellanox and Intel both produce software stacks for their products that typically include their own customized versions of OFED.

112 questions
0
votes
1 answer

Infiniband P_Keys and the Linux kernel

I'm trying to configure Infiniband partitions between two Debian based Linux hosts running the 4.15 kernel and A Mellanox SX6036 switch. I've setup a "DMZ" partition on the swich using a PKey of 0x0001 and added the Port GUID numbers from the active…
Server Fault
  • 3,714
  • 12
  • 54
  • 89
0
votes
0 answers

QLogic 12300 :: Can't use more than 6 ports

I have 2 QLogic 12300 InfiniBand switches (12300-BS01) with 36 ports. For some strange reason I am unable to connect anything to ports 7-36, it seems that only ports 1-6 are working. I Googled and found that in 12300 16-port switch (12300-BS16),…
0
votes
1 answer

MPI can not find Infiniband interface point-to-point

I think I have a Infiniband setup issue. I'm not very experienced with setting up Infiniband or networks. If I try to force MPICH to use IB, I get errors: [Bryan@node1 shared]$ ./mpich-3.3.1/bin/mpiexec -hosts=node1,node2 -iface=ib0 -n 4…
0
votes
1 answer

Can you mix 10gb and Infiniband for iScsi storage target

I'm setting up a Fedora server to act as an iScsi storage target. The primary storage network will be 25gb Infiniband. Is it possible and/or safe to also allow the target to be accessed by 10gb ethernet concurrently ?
bumble_bee_tuna
  • 443
  • 11
  • 26
0
votes
1 answer

HP Infiniband HCA on Fujitsu Motherboard

I am building a 2 node cluster using node-to-node Infiniband connection. I have HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP (P/N : 649282-B21) and Fujitsu D3118-A22 GS-2 Mainboard with dual Xeon E5-2690s and Ubuntu 16.04.4. The problem is…
0
votes
1 answer

Can not setup Mellanox connectx-2 in 40g speed

I have Mellanox connectX-2 network card (MT26428) and I installed MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu16.04-x86_64 driver from Mellanox repository but I'm wondering this equipment setup 20G at maximum although I expected it to setup 40G instead. here…
MDK
  • 101
  • 2
0
votes
1 answer

Any way to segregate ports on a Mellanox Infiniband switch?

I currently have two infiniband IPoIB networks across multiple unmanaged switches (along with some PCs running as the subnet managers) with multiple Linux nodes attached with Infiniband adapters. There is an SX6036 switch in-house that I'd like to…
Server Fault
  • 3,714
  • 12
  • 54
  • 89
0
votes
1 answer

What is a 4x 10gbe QSFP-QSFP cable?

I was looking for QSFP cables on Ebay and came across a Brocade 58-0000033-01 marked as 4x 10GE even though it has one QSFP connector on each end. I don't understand what this cable is for, will it work with a 10GBE/40GB Infiniband HCA?
chew socks
  • 212
  • 2
  • 8
0
votes
1 answer

Managed switch for Mellanox MCX354A-FCBT cards?

I've inherited two Mellanox Infiniband environments. One on a LAN, and the other in a DMZ. Both IB networks have their own SX6005 unmanaged switches and a couple small desktop computers running the subnet manager for the network. As I understand,…
Server Fault
  • 3,714
  • 12
  • 54
  • 89
0
votes
1 answer

I want to connect to bridge using infiniband

I want to connect the physical port (ib0) to the bridge (bro). I am constructing kvm, but the physical port of infiniband to the virtual bridge I want to connect, but I could not do it well. Please tell me who understands. OS: centos7 I did: nmcli…
keiji
  • 11
0
votes
1 answer

How to set Infiniband port from INIT to ACTIVE

I have the following setup. 7 Nodes, let's say they are called gauss1 to gauss7. I have a stable connection between gauss1 to gauss6. Just gauss7 is making trouble. # ibnodes Ca : 0x0002c90300f2eef0 ports 2 "gauss1 mlx4_0" Ca : 0x0002c90300f2ef20…
Shibumi
  • 151
  • 1
  • 6
0
votes
1 answer

Ethernet switch with Infiniband uplink

There are Ethernet switches (copper based, i.e. with RJ45 sockets) that have Fibre uplink. Example: Netgear GS110TP Are there any such switches with Infiniband uplinks? If not then why?
Greg
  • 198
  • 10
0
votes
0 answers

Ideal configuration for a head node?

Which hardware should I concentrate on, when assembling a head node for an HPC cluster? The main task for the head node is to relay instructions to the compute nodes which will be running artificial intelligence algorithms. Ubuntu 14.04 LTS will be…
Rushat Rai
  • 111
  • 4
0
votes
1 answer

If I replace an Infiniband switch should I restart opensm?

I have been using Mellanox ConnectX 2 QDR cards in a point to point link between two machines, with opensm running on one machine. This has worked very well. Recently I got a Mellanox IS5022 switch - which does not run an SM. So I know I still need…
Eric M
  • 113
  • 5
0
votes
1 answer

Troubles with ibverbs in OpenSUSE Leap 42.2 for Mellanox ConnectX-3 HCA

SOLVED I got troubles setting up infiniband software from OpenSUSE repository. The same card works fine with SLES12 + Mellanox OFED stack. Installing everything containing 'infiniband' from yast I see that HCA is up and diagnostic tools like ibnodes…
catbus
  • 1
  • 2