4

I've started working through a snakemake tutorial and the very first workflow from there does not work. Here is the rule I am using:

rule make_a_copy:
    input:
        "a.txt"
    output:
        "a_copy.txt"
    shell:
        """
        copy {input} {output}
        """

Then I run the workflow with

snakemake -p a_copy.txt

This results in the following output:

Building DAG of jobs...
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   convert_to_upper_case
    1

[Thu Nov 21 15:33:18 2019]
rule convert_to_upper_case:
    input: a.txt
    output: a_copy.txt
    jobid: 0


        copy a.txt a_copy.txt

Waiting at most 5 seconds for missing files.
MissingOutputException in line 1 of D:\OneDrive\projects\reproducible_research_course\snakemake\Snakefile:
Missing files after 5 seconds:
a_copy.txt
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: D:\OneDrive\projects\reproducible_research_course\snakemake\.snakemake\log\2019-11-21T153318.425144.snakemake.log

If I run copy a.txt a_copy.txt in cmd (I'm on windows) the command does produce an a.upper.txt file.

What am I missing?

Evgenii
  • 335
  • 2
  • 11
  • 1
    Is ` command` the same as `command` in windows? – The Unfun Cat Nov 21 '19 at 14:17
  • 1
    Try increasing `--latency-wait`? Doubt this is the fix though. – Manavalan Gajapathy Nov 21 '19 at 19:09
  • @TheUnfunCat, yes, I tried running ` copy a.txt a.upper.txt` and it worked. – Evgenii Nov 21 '19 at 19:46
  • @ManavalanGajapathy, the command takes microseconds so 5 seconds should certainly be enough. But just to be sure I did try setting it to 30 seconds. No luck, unfortunately. – Evgenii Nov 21 '19 at 19:48
  • 1
    I tested your code just to be certain, and it works fine on mac. Not a surprise. Can't comment on Windows as I never used snakemake in it. – Manavalan Gajapathy Nov 21 '19 at 21:56
  • 1
    Have you tried changing the triple quotes around the shell command to just single quotes? – Colin Nov 22 '19 at 00:15
  • @ManavalanGajapathy, thank you for checking on Mac. Now I am sure it is Windows-related. – Evgenii Nov 22 '19 at 08:44
  • @Colin, changing it to a one-line single-quoted string helped! Since it did not make any sense to me I tried a one-line triple-quoted string - and it worked as well! It turns out the problem was not with the triple quotes per se but with the string starting with an empty line. Thank you for leading me to the solution. – Evgenii Nov 22 '19 at 09:03
  • 2
    @Eugene - nice detective work! I would suggest posting your solution as an answer rather than as an edit to your question. Also, I wonder whether this behaviour should be reported to the Snakemake issue tracker... – dariober Nov 22 '19 at 09:48
  • I posted my solution as an answer. – Evgenii Nov 23 '19 at 05:27
  • 1
    @dariober, about the issue tracker: now that I know the answer it was easy to find a 2013(!) issue posted [here](https://bitbucket.org/snakemake/snakemake/issues/54/windows-support). The author's conclusion was that since shell commands are mostly not portable anyways it is not something that snakemake should care about. – Evgenii Nov 23 '19 at 05:31

1 Answers1

4

Thanks to the comments from @the-unfun-cat and @Colin, I figured out the problem. Only the first line of the shell command string is executed. So, these will work:

shell:
    "copy a.txt a_copy.txt"

shell:
    "    copy a.txt a_copy.txt    "

shell:
    """copy a.txt a_copy.txt"""

But 

shell:
    """
    copy a.txt a_copy.txt
    """

which is equivalent to

shell:
    "\n    copy a.txt a_copy.txt"

will not work since only the first line (which is empty) will be executed.

Here is a bitbucket issue where this very problem was discussed in 2013. The snakemake's author suggested using run instead of shell:

rule:
  ...
  run:
    shell(" & ".join(xx))

This, of course, makes the code not portable but he argues that shell commands are not portable anyways.

Evgenii
  • 335
  • 2
  • 11