3

I am trying to use Snakemake on AWS EKS, running shell commands in singularity containers. If I understood correctly, Snakemake is then run as a container itself by kubernetes, so I would run a singularity container inside a Pod. I have a ubuntu image in the same folder of the Snakefile. How do I tell Singularity the path where the image is stored?

I have tried to run the workflow specifying an image in the DockerHub and it works perfectly. However I cannot make it work with images stored in my EC2 instance, if I specify --kubernetes flag (if I run it without --kubernetes, it runs perfectly also with local images). I tried specifying the path of the image on the EC2 instance, but it is not working.

My Snakefile looks something like that:

rule test:
    singularity:
        "ubuntu.simg"
    output:
        "out.txt"
    shell:
        "cat /etc/os-release > {output}"

and I run Snakemake with the command:

snakemake --kubernetes --default-remote-provider S3 --default-remote-prefix S3-bucket-name --use-singularity

Desired output is the file "out.txt" created in the specified s3 bucket with the info about the os release of the "ubuntu.sigm" container. The actual result is a failed job. If I inspect with

kubectl logs snakejob-c25eaf1f-6ad4-505e-94a0-646543a59d33

I get the error:

ERROR  : Image path ubuntu.simg doesn't exist: No such file or directory
ABORT  : Retval = 255

Anyone who had the same issue?

meo
  • 31
  • 1
  • It looks like the environment inside the kubernetes is not what you're expecting to be there, or at least in a different directory than you're expecting. What happens if you remove the `singularity` part of the rule, and change the shell content to `echo $PWD; ls -la`? – tsnowlan Jul 02 '19 at 08:41
  • I get `/workdir total 0 drwxr-xr-x 4 root root 94 Jul 2 12:39 . drwxr-xr-x 1 root root 46 Jul 2 12:39 .. drwxr-xr-x 2 root root 23 Jul 2 12:39 ..2019_07_02_12_39_31.635282758 lrwxrwxrwx 1 root root 31 Jul 2 12:39 ..data -> ..2019_07_02_12_39_31.635282758 drwxr-xr-x 9 root root 113 Jul 2 12:39 .snakemake lrwxrwxrwx 1 root root 16 Jul 2 12:39 Snakefile -> ..data/Snakefile [Tue Jul 2 12:39:32 2019] Finished job 0. 1 of 1 steps (100%) done Complete log: /workdir/.snakemake/log/2019-07-02T123932.011750.snakemake.log` – meo Jul 02 '19 at 12:47
  • Formatting gets a little wonky in these comments, but you may also want to look into the `..data` and `.snakemake` directories. If the image is still not there, it looks like the bucket content is not getting loaded into the pod. That could be from the pods not having the correct permissions or needing additional params. – tsnowlan Jul 03 '19 at 09:52
  • The image is not in ..data nor in .snakemake, nor in any other subfolder. When the image gets pulled automatically from DockerHub (named docker://image_name) I see a new image inside the pod in .snakemake/singularity. If however I try to specify that image in the input singularity field with `singularity: "/workdir/.snakemake/image_name.simg" ` I get the same error as above. – meo Jul 19 '19 at 09:03

0 Answers0