3

I have a lot of samples that go through a process which sometimes fail (deterministically). In such a case, I would want the failing process to stop, but all other samples to still get submitted and processed independently.

If I understand correctly, setting errorStrategy 'ignore' will continue the script within the failing process, which is not what I want. And errorStrategy 'finish' would stop submitting new samples, even though there is no reason for the other samples to fail too. And while errorStrategy 'retry' could technically work (by repeating the failing processes while the good ones get through), that doesn't seem like a good solution.

Am I missing something?

Alexlok
  • 2,999
  • 15
  • 20

1 Answers1

4

If a process can fail deterministically, it might be better to handle this situation somehow. Setting the errorStrategy directive to 'ignore' will mean any processes execution errors are ignored and allow your workflow continue. For example, you might get a process execution error if a process exits with a non-zero exit status or if one or more expected output files are missing. The pipeline will continue, however downstream processes will not be attempted.

Contents of test.nf:

nextflow.enable.dsl=2

process foo {

    tag { sample }

    input:
    val sample

    output:
    path "${sample}.txt"

    """
    if [ "${sample}" == "s1" ] ; then
        (exit 1)
    fi
    if [ "${sample}" == "s2" ] ; then
        echo "Hello" > "${sample}.txt"
    fi
    """
}

process bar {

    tag { txt }

    input:
    path txt

    output:
    path "${txt}.gz"

    """
    gzip -c "${txt}" > "${txt}.gz"
    """
}

workflow {

    Channel.of('s1', 's2', 's3') | foo | bar
}

Contents of nextflow.config:

process {

  // this is the default task.shell:
  shell = [ '/bin/bash', '-ue' ]

  errorStrategy = 'ignore'
}

Run with:

nextflow run -ansi-log false test.nf

Results:

N E X T F L O W  ~  version 20.10.0
Launching `test.nf` [drunk_bartik] - revision: e2103ea23b
[9b/56ce2d] Submitted process > foo (s2)
[43/0d5c9d] Submitted process > foo (s1)
[51/7b6752] Submitted process > foo (s3)
[43/0d5c9d] NOTE: Process `foo (s1)` terminated with an error exit status (1) -- Error is ignored
[51/7b6752] NOTE: Missing output file(s) `s3.txt` expected by process `foo (s3)` -- Error is ignored
[51/267685] Submitted process > bar (s2.txt)
Steve
  • 51,466
  • 13
  • 89
  • 103
  • I see, thank you! And if I'm not mistaken, the failing process (foo on s1 here) is guaranteed to stop at the first error because of the automatic `set -e`? In the case when the failure is not an `exit 1` but an error in an external program (so `foo` could continue running if there is no `set -e`). – Alexlok Oct 07 '21 at 00:18
  • 1
    @Alexlok Correct: `set -e` will exit immediately if a command exits with a non-zero status. I.e. Commands following this will not be run. Note that if you are piping commands together in your script, there's no guarantee unless you use `shell = [ '/bin/bash', '-euo', 'pipefail' ]`. Also note that I put `exit 1` inside of a subshell to simulate an external command. Without `set -e`, `foo` will continue and process s1 will fail in the same way as s3 does (i.e. Missing output files...) – Steve Oct 07 '21 at 01:14
  • Sorry, one aspect is still not fully clear to me: in your example, you added `set -ue` in the config file; iirc Nextflow automatically adds it to every file that gets run. And, (it seems to me through experimenting) the `errorStrategy 'ignore'` does not change that behavior? – Alexlok Oct 07 '21 at 14:45
  • 1
    @Alexlok No worries - I was just pointing out what the default shell is. The value of `process.shell` in the nextflow.config just becomes the shebang that gets used in each of the workdir `.command.sh` scripts. Your last point is also correct: setting `errorStrategy 'ignore'` will not change how the script is run. If the shebang is `#!/bin/bash -e`, i.e. `set -e`, the command script will still exit immediately if a command exits with a non-zero exit status, regardless of the errorStrategy. Note that you could do a `set +e` before your sometimes failing command to override it. – Steve Oct 07 '21 at 15:17
  • 1
    Makes sense, thank you! – Alexlok Oct 07 '21 at 15:42