1

I have a Snakefile that runs a python script which outputs many files in a directory. I wrote the following Snakefile script to execute this

MODELS = ["A", "B", "C"]
SEEDS = [1, 2, 3]

rule all: 
    input:
        expand("outputs/{model}/seed{seed}", model=MODELS, seed=SEEDS)

rule sub: 
    input: 
        {model}.py
    output: 
        directory("outputs/{model}/seed{seed}")
    run: 
        command = "python3 {} --seed {}".format(input, wildcards.seed)
        shell(command)

Each python script files A.py, B.py, and C.py executes for hours. I want to be able to use sbatch that submits job without waiting for it to finish executing.

$ snakemake --cluster "sbatch --job-name=snakemake" --jobs 200 --latency-wait 1000

When I execute the following command, some files do not get run and Snakemake does not terminate. I tried writing a bash script that contains the above snakemake command and executed sbatch script.sh but that did not submit jobs in the Snakefile.

Is there a way to do this without snakemake waiting for sbatch jobs to finish executing?

milkyway42
  • 144
  • 1
  • 6
  • I usually use your second approach: `sbatch script.sh` where script.sh runs snakemake and spawns new slurm tasks for each step. If you give more details about how that went wrong (script contents, logs, etc), I might be able to help. – Marmaduke Feb 22 '21 at 19:28

1 Answers1

0

It would not be a perfect solution, but have you tried the --immediate-submit flag? It will just submit all jobs to slurm with no regard for dependencies and without waiting. Normally, you'd supply a wrapper for sbatch that would translate snakemake dependencies to SLURM dependencies, but for a simple workflow like this (ie only one rule run many times), you might get away without that.

Marmaduke
  • 541
  • 3
  • 8