0

I would like to know how to invoke InfiniBand hardware on CentOS 6x cluster with OpenMPI (static libs.) for running my C++ code. This is how I compile and run:

/usr/local/open-mpi/1.10.7/bin/mpic++ -L/usr/local/open-mpi/1.10.7/lib -Bstatic main.cpp -o DoWork

usr/local/open-mpi/1.10.7/bin/mpiexec -mca btl tcp,self --hostfile hostfile5 -host node01,node02,node03,node04,node05 -n 200 DoWork

Here, "-mca btl tcp,self" reveals that TCP is used, and the cluster has InfiniBand.

What should be changed in compiling and running commands for InfiniBand to be invoked? If I just replace "-mca btl tcp,self" with "-mca btl openib,self" then I get plenty of errors with relevant one saying:

At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL.

Thanks very much!!!

Boki
  • 193
  • 1
  • 3
  • 15
  • 1
    Remove the `-mca btl ...` parameter. Open MPI is modular and it automatically picks up the best communication interface. If there is a usable InfiniBand hardware that Open MPI can detect, it will automatically use the `openib` module since it has much higher precedence than the TCP module. – Hristo Iliev Jul 13 '17 at 20:27
  • Thanks very much for the reply and help, Hristo. I have tried following: **usr/local/open-mpi/1.10.7/bin/mpiexec --hostfile hostfile5 -host node01,node02,node03,node04,node05 -n 200 DoWork** and it didn't work, unfortunately. – Boki Jul 13 '17 at 21:25
  • What exactly didn't work? – Hristo Iliev Jul 14 '17 at 07:35
  • Here is the error message: -------------------------------------------------------------------------- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: node02 Local device: mlx5_1 Local port: 1 CPCs attempted: rdmacm, udcm -------------------------------------------------------------------------- – Boki Jul 14 '17 at 18:07
  • If I check InfiniBand settings with "ibhost" command, I get following errors: -bash-4.1$ ibhosts ibwarn: [168793] mad_rpc_open_port: client_register for mgmt 1 failed src/ibnetdisc.cand:766; can't open MAD port ((null):0) /usr/sbin/ibnetdiscover: iberror: failed: discover failed – Boki Jul 14 '17 at 18:09
  • The latter message means that the `ib_mad` (or was it `ib_umad`?) kernel module is not loaded, so the MAD interface is missing. It is an InfiniBand software configuration problem, so not really for Stack Overflow. Try on [Super User](https://superuser.com/) or [Server Fault](https://serverfault.com/). – Hristo Iliev Jul 14 '17 at 22:48
  • Thanks Hristo. I'll see to talk to our "root" next week to ask if they can properly install ib. – Boki Jul 14 '17 at 23:32
  • Here is addtional info, I appreciate if you can comment. I did ibstat on the login node and among the large output, here are the important four lines: mlx5_ 0: Physical state: LinkUp mlx5_ 1: Physical state: LinkUp mlx5_ 2: Physical state: Disabled mlx5_ 3: Physical state: Disabled – Boki Jul 14 '17 at 23:34
  • What do you think? – Boki Jul 14 '17 at 23:35
  • Does this definitely show that on this node the two out of four processors are not having InfiniBand connection? There are two sockets, and each socket holds two processors (each processor 10 cores, each core 2 logical threads). So, one socket (two processors) is not connected with InfiniBand. – Boki Jul 15 '17 at 00:05
  • No, it's a 4-port InfiniBand HCA with only two ports connected. It doesn't limit the CPUs that have access to it in any way. – Hristo Iliev Jul 15 '17 at 08:02
  • I was told by the "root" that in this machine is RoCE (RDMA over Converged Ethernet) with 100Gb/s. – Boki Jul 17 '17 at 16:27
  • I guess RoCE gets automatically used when I use mpiexec with " -mca btl tcp,self" params. – Boki Jul 17 '17 at 16:28
  • Consult the Open MPI FAQ on [running it over RoCE](http://www.open-mpi.de/faq/?category=openfabrics#ompi-over-roce). The crucial parameter is probably the one that enables the `rdmacm` CPC: `--mca btl_openib_cpc_include rdmacm`. – Hristo Iliev Jul 19 '17 at 07:38

0 Answers0