How could I run Open MPI under Slurm

Question

I am unable to run Open MPI under Slurm through a Slurm-script.

In general, I am able to obtain the hostname and run Open MPI on my machine.

$ mpirun hostname
myHost
$ cd NPB3.3-SER/ && make ua CLASS=B && mpirun -n 1 bin/ua.B.x inputua.data # Works

---

But if I do the same operation through the slurm-script mpirun hostname returns empty string and consequently I am unable to run mpirun -n 1 bin/ua.B.x inputua.data.

slurm-script.sh:

#!/bin/bash
#SBATCH -o slurm.out        # STDOUT
#SBATCH -e slurm.err        # STDERR
#SBATCH --mail-type=ALL

export LD_LIBRARY_PATH="/usr/lib/openmpi/lib"
mpirun hostname > output.txt # Returns empty
cd NPB3.3-SER/ 
make ua CLASS=B 
mpirun --host myHost -n 1 bin/ua.B.x inputua.data

$ sbatch -N1 slurm-script.sh
Submitted batch job 1

---

The error I am receiving:

There are no allocated resources for the application
  bin/ua.B.x
that match the requested mapping:    
------------------------------------------------------------------
Verify that you have mapped the allocated resources properly using the
--host or --hostfile specification.

A daemon (pid unknown) died unexpectedly with status 1 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
------------------------------------------------------------------
An ORTE daemon has unexpectedly failed after launch and before
communicating back to mpirun. This could be caused by a number
of factors, including an inability to create a connection back
to mpirun due to a lack of common network interfaces and/or no
route found between them. Please check network connectivity
(including firewalls and network routing requirements).
------------------------------------------------------------------

Answer 1

If Slurm and OpenMPI are recent versions, make sure that OpenMPI is compiled with Slurm support (run ompi_info | grep slurm to find out) and just run srun bin/ua.B.x inputua.data in your submission script.

Alternatively, mpirun bin/ua.B.x inputua.data should work too.

If OpenMPI is compiled without Slurm support the following should work:

srun hostname > output.txt
cd NPB3.3-SER/ 
make ua CLASS=B 
mpirun --hostfile output.txt -n 1 bin/ua.B.x inputua.data

Make sure also that by running export LD_LIBRARY_PATH="/usr/lib/openmpi/lib" you do not overwrite other library paths that are necessary. Better would probably be export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/lib/openmpi/lib" (or a more complex version if you want to avoid a leading : if it were initially empty.)

Answer 2

What you need is: 1) run mpirun, 2) from slurm, 3) with --host. To determine who is responsible for this not to work (Problem 1), you could test a few things. Whatever you test, you should test exactly the same via command line (CLI) and via slurm (S). It is understood that some of these tests will produce different results in cases CLI and S.

A few notes are: 1) You are not testing exactly the same things in CLI and S. 2) You say that you are "unable to run mpirun -n 1 bin/ua.B.x inputua.data", while the problem is actually with mpirun --host myHost -n 1 bin/ua.B.x inputua.data. 3) The fact that mpirun hostname > output.txt returns an empty file (Problem 2) does not necessarily have the same origin as your main problem, see paragraph above. You can overcome this problem by using scontrol show hostnames or with the environment variable SLURM_NODELIST (on which scontrol show hostnames is based), but this will not solve Problem 1.

---

To work around Problem 2, which is not the most important, try a few things via both CLI and S. The slurm script below may be helpful.

#SBATCH -o slurm_hostname.out        # STDOUT
#SBATCH -e slurm_hostname.err        # STDERR
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:+${LD_LIBRARY_PATH}:}/usr/lib64/openmpi/lib"

mpirun hostname > hostname_mpirun.txt               # 1. Returns values ok for me
hostname > hostname.txt                             # 2. Returns values ok for me
hostname -s > hostname_slurmcontrol.txt             # 3. Returns values ok for me
scontrol show hostnames > hostname_scontrol.txt     # 4. Returns values ok for me
echo ${SLURM_NODELIST} > hostname_slurmcontrol.txt  # 5. Returns values ok for me

(for an explanation of the export command see this). From what you say, I understand 2, 3, 4 and 5 work ok for you, and 1 does not. So you could now use mpirun with suitable options --host or --hostfile.

Note the different format of the output of scontrol show hostnames (e.g., for me cnode17<newline>cnode18) and echo ${SLURM_NODELIST} (cnode[17-18]).

The host names could perhaps also be obtained in file names set dynamically with %h and %n in slurm.conf, look for e.g. SlurmdLogFile, SlurmdPidFile.

---

To diagnose/work around/solve Problem 1, try mpirun with/without --host, in CLI and S. From what you say, assuming you used the correct syntax in each case, this is the outcome:

1. mpirun, CLI (original post). "Works".

2. mpirun, S (comment?). Same error as item 4 below? Note that mpirun hostname in S should have produced similar output in your slurm.err.

3. mpirun --host, CLI (comment). Error

   There are no allocated resources for the application bin/ua.B.x that match the requested mapping:
   ...
   This may be because the daemon was unable to find all the needed shared
   libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
   location of the shared libraries on the remote nodes and this will
   automatically be forwarded to the remote nodes.
   

4. mpirun --host, S (original post). Error (same as item 3 above?)

   There are no allocated resources for the application
     bin/ua.B.x
   that match the requested mapping:    
   ------------------------------------------------------------------
   Verify that you have mapped the allocated resources properly using the
   --host or --hostfile specification.
   ...
   This may be because the daemon was unable to find all the needed shared
   libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
   location of the shared libraries on the remote nodes and this will
   automatically be forwarded to the remote nodes.
   

As per comments, you may have a wrong LD_LIBRARY_PATH path set. You may also need to use mpi --prefix ...

Related? https://github.com/easybuilders/easybuild-easyconfigs/issues/204