Error submitting jobscript (exit code 127) when invoking snakemake through cron job on SLURM cluster

Question

I'm trying to run a snakemake pipeline through crontab on a SLURM cluster. Here is the bash script that I used to send to the slurm.

#!/bin/bash
#SBATCH --job-name=nextstrain

snakemake --configfile config.yaml --jobs 100 --keep-going --rerun-incomplete --latency-wait 360 --cluster 'sbatch' -r -p --useconda

This scrip runs as intended. However, when I run the script through crontab as so:

0 8 * * 1 /bin/bash /home/user/snakemake_automate.sh

I get the error:

Error submitting jobscript (exit code 127):

I am not sure what I should do to fix this error.

Answer 1

Exit code 127 means command not found I suspect you need to load a module or conda env prior to invoking snakemake. When you run the script interactively it will use your current environment, but through cron it may not source your bashrc or similar.

Answer 2

Your error implies cron can't find sbatch, which snakemake uses to submit job scripts to the SLURM cluster.

1. Find where sbatch resides by running which -a sbatch. This might give you:

   $ which -a sbatch
   /usr/local/bin/sbatch
   /usr/bin/sbatch
   

2. Prepend that path to your PATH variable in the crontab before calling snakemake:

   export PATH=/usr/bin/:$PATH
   /bin/bash /home/user/snakemake_automate.sh
   

   or in the bash script itself (remember the PATH variable that cron sees may be very different than what you as a local user see):

   #!/bin/bash
   #SBATCH --job-name=nextstrain
   export PATH=/usr/bin/:$PATH
   snakemake --configfile config.yaml --jobs 100 --keep-going --rerun-incomplete --latency-wait 360 --cluster 'sbatch' -r -p --useconda
   

3. To get more error details, redirect snakemake outputs to a log file in your crontab command:

   0 8 * * 1 /bin/bash /home/user/snakemake_automate.sh > /home/user/snakemake.log 2>&1
   

4. It could also be helpful to set -xeuo pipefail in your shell script for more verbose output.
5. You can also add extra arguments for snakemake, like -p to print shell commands, --debug, and --verbose.