Snakemake explicit handling for Out Of Memory (OOM) failures

Question

A Snakemake workflow can re-attempt for each restart after any type of failure, including if the error is of an Out Of Memory (OOM) doing e.g.

def get_mem_mb(wildcards, attempt):
    return attempt * 100

rule:
    input:    ...
    output:   ...
    resources:
        mem_mb=get_mem_mb
    shell:
        "..."

Is there anyway in Snakemake to deal explicitly with a memory-related error, as NextFlow does. e.g. when Exit error is memory related (137 in a LSF system)?

process foo {

    memory { 2.GB * task.attempt }
    time { 1.hour * task.attempt }

    errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
    maxRetries 3

    script:
    <your job here>

}

I could not find this information anywhere,

thanks

score 1 · Answer 1 · answered Oct 20 '21 at 21:33

I am not sure if there is an explicit way for Snakemake to handle out of memory errors. However, the memory function you have in your code example is what I've done to handle memory issues using Snakemake.

To make use of the function, you need to provide the --rerun-incomplete option when executing Snakemake, so that failed jobs will be rerun. You can control the number of times Snakemake will retry with --restart-times.

Snakemake explicit handling for Out Of Memory (OOM) failures

1 Answers1