5

I am currently working on singularity and docker. I am running singularity for MPI environment.

I want to use advantages of singularity for MPI but singularity files are very large. So after running the singularity image I want to convert it to docker image and then save it which will save the disk space.

Is this possible to convert singularity image into docker image?

jonrsharpe
  • 115,751
  • 26
  • 228
  • 437
alex
  • 303
  • 1
  • 4
  • 6

3 Answers3

4

As @tsnowlan said in their answer, typically the workflow is from Docker to Singularity. But there is a way to make a Docker image from an existing Singularity image. This would not make use of Docker's layer caching features.

The general idea is to:

  1. Dump the Singularity image filesystem as a squashfs file.
  2. Extract the squashfs file into a directory.
  3. Create a Dockerfile that inherits from scratch, copies over the Singularity image's filesystem, and sets environment variables and other things.
  4. Build the Docker image.

Here it is in bash, demonstrated on alpine:latest:

singularity pull docker://alpine:latest
# Find out which SIF ID to use (look for Squashfs)
singularity sif list alpine_latest.sif
# Get the environment variables defined in the Singularity image.
singularity sif dump 2 alpine_latest.sif
singularity sif dump 3 alpine_latest.sif > data.squash
unsquashfs -dest data data.squash
# See the Dockerfile definition below
docker build --tag alpine:latest .

Contents of Dockerfile:

FROM scratch
COPY data /
ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CMD ["/bin/ash"]

For more information on Singularity and Docker, I recommend looking at Singularity's documentation on the topic.

jkr
  • 17,119
  • 2
  • 42
  • 68
0

Containerizing MPI applications is not terribly difficult with Singularity, but it comes at the cost of additional requirements for the host system.

This means you have to choose the right image base for this custom image. Something like this example : 

FROM tacc/tacc-ubuntu18-mvapich2.3-psm2:0.0.2

RUN apt-get update && apt-get upgrade -y && apt-get install -y python3-pip
RUN pip3 install mpi4py

COPY pi-mpi.py /code/pi-mpi.py
RUN chmod +x /code/pi-mpi.py

ENV PATH "/code:$PATH"
Abdennour TOUMI
  • 87,526
  • 38
  • 249
  • 254
  • Containerizing MPI applications with singularity is not my concern. I have done it previously. I just wanted to convert singularity image into docker image. – alex Mar 02 '20 at 05:59
0

Generally speaking, it's done in the other direction: building a singularity image based on Docker. If you plan on storing it as a docker image rather than singularity, your best bet would be to build it as Docker and convert to singularity as needed. I know of several labs using this method: build via Dockerfile, push to Docker hub for storage, build singularity image directly from docker hub image.

However, singularity images are usually smaller than docker images. The base docker image mentioned in the other answer, tacc/tacc-ubuntu18-mvapich2.3-psm2:0.0.2, is 497MB but the singularity image is only 160MB. ubuntu:latest is 64.2MB vs 25MB, python:latest is 933MB vs 303MB.

tsnowlan
  • 3,472
  • 10
  • 15
  • singularity images are not typically smaller. the sizes for the docker images you are quoting are the uncompressed sizes. the compressed size for `ubuntu:latest`, for example, is 25.9 MB, and python is 341.02 MB. – jkr Mar 03 '20 at 00:52
  • Glad to see docker can compress the images much smaller. While that does make them roughly comparable in size, singularity is still a bit smaller in all those cases. – tsnowlan Mar 03 '20 at 08:45
  • Yes docker uses gzip compression, and squashfs uses gzip compression by default. singularity uses squashfs under the hood. – jkr Mar 03 '20 at 14:05