How to prevent snakemake from stopping after failed test suite?

Question

I am attempting to automate integration testing with snakemake (I need the output from some of the files so this seemed like a good tool). However when I need to run two test suites in pytest, the workflow stops if a single test in either suite fails. So for example I have:

rule run_tests:
    run:
        commands = [
            "pytest test_that_should_fail",
            "pytest test_that_should_succeed"
        ]
        for c in commands:
            shell(c)

And I need the output of the test that should fail for the latter test. Is there a way to prevent snakemake from stopping after running "pytest test_that_should_fail" ? Additionally snakemake stops without any sort of error message just a generic: "Error in rule run_tests: jobid: 0"

Does using `--verbose` on snakemake command help to show the output of the commands? — Maicon Mauricio, May 04 '21 at 23:46

Maicon Mauricio · Accepted Answer · 2021-05-05T16:35:09.337

2

You could write something like:

commands = [
   "pytest test_that_should_fail || true",
   "pytest test_that_should_succeed || true",
]

Then the shell commands will have exit code 0, which means succes, thus proceeding to execute other commands.

EDIT:

This is a simple way to solve the problem, not the cleanest nor the most idiomatic as pointed out by @bli.

edited May 05 '21 at 16:35

answered May 04 '21 at 23:33

Maicon Mauricio

2,052
1
13
29

score 1 · Answer 2 · answered May 05 '21 at 07:46

I would suggest you let Snakemake deal with the different tests by running only one test per rule instance.

This could be done as follows (not tested):

TESTS = ["test_that_should_fail", "test_that_should_succeed"]

rule all:
    input:
        logs = expand("{test}.log", test=TESTS),
        errs = expand("{test}.err", test=TESTS),

rule run_test:
    output:
        log = "{test}.log",
        err = "{test}.err",
    shell:
        """
        pytest {wildcards.test} 1> {output.log} 2> {output.err}
        """

Then, you can use option -k of snakemake to be sure all tests will be run. But it would probably be cleaner to use || true (as suggested in another answer) if a non-zero return code is something expected.

How to prevent snakemake from stopping after failed test suite?

2 Answers2