4

Background: I have successfully created MPI-based applications within a Singularity container and executed them using the hybrid mode recommended by the Singularity documentation. This requires that I have a "compatible" version of MPI installed both in the container and on the host O/S. I understand why this is necessary if I will be runnings jobs across multiple nodes.

But in our use case we have slow interconnects and so it pretty much never makes sense for us to run jobs over more than one node. Hybrid mode forces me to keep track of every version of MPI that has been built into every container that might ever be used to make sure that the right MPI is available on the host O/S. The whole reason I've gone to containers in the first place is to avoid having to deal with this kind of version-dependency hassle.

My question: If I am OK with the limitation that MPI applications can only be executed within a single physical node, is there a way for me to build an MPI application in a Singularity container that is not dependent on having anything installed on the host O/S?

bk_biophysics
  • 41
  • 2
  • If you use SLURM, an alternative is to direct launch (aka `srun`) instead of `mpirun`. – Gilles Gouaillardet Feb 10 '20 at 13:37
  • Or you might consider running `mpirun` inside the container. – Gilles Gouaillardet Feb 10 '20 at 14:05
  • Thank you @GillesGouaillardet, I'll look into srun -- I've not used this before. Just doing a plain singularity run as I might do for a shared-memory or serial job but using mpirun within the container does not work. I get the following error: HYDU_create_process (utils/launch/launch.c:75): execvp error on file srun (No such file or directory) – bk_biophysics Feb 10 '20 at 18:59
  • Ahh, so maybe I have a SLURM issue? If I login directly to a compute node and do singularity run ... that works. But if I put that identical line within a script and execute it via sbatch I get the error above. So maybe I'm really asking for help here with SLURM. – bk_biophysics Feb 10 '20 at 19:20
  • Note that within the SLURM script I have checked the path and even done 'which srun' and those look OK. So it is definitely that case that srun is on the path in the environment in which I'm executing mpirun. – bk_biophysics Feb 10 '20 at 19:39
  • since you are already using SLURM, you have to use `srun` instead `mpirun` inside your script (e.g. **not** in your singularity container), note you might have to make sure SLURM is using `PMIx` so you do not need your PMI library inside the containers matches the SLURM version of your host. If you want to use `mpirun` inside your container, you should direct it **not** to use SLURM (I am not familiar with MPICH, but there might be a command line to tell it not to use SLURM, or you might remove some `SLURM_*` environment variable before invoking `mpirun` in your container). – Gilles Gouaillardet Feb 11 '20 at 03:34

1 Answers1

0

I'm not the strongest on MPI, but if you're not trying to communicate with other nodes (or jobs spawned by other processes on the same compute node) you should be able to use singularity exec mpi_image.sif mpirun ....

It may also be useful to modify your Singularity definition to be able to run the image as an instance. It makes singularity behave more like Docker, allowing you to singularity shell into a running execution environment. This greatly simplifies debugging when doing things that are not documented very well.

tsnowlan
  • 3,472
  • 10
  • 15