Does snakemake support none file Input?

Question

I get a MissingInputException when I run the following rule:

configfile: "Configs.yaml"

rule download_data_from_ZFIN:
    input:
        anatomy_item = config["ZFIN_url"]["anatomy_item"],
        xpat_stage_anatomy = config["ZFIN_url"]["xpat_stage_anatomy"],
        xpat_fish = config["ZFIN_url"]["xpat_fish"],
        anatomy_synonyms = config["ZFIN_url"]["anatomy_synonyms"]
    output:
        anatomy_item = os.path.join(os.getcwd(), config["download_data_from_ZFIN"]["dir"], "anatomy_item.tsv"),
        xpat_stage_anatomy = os.path.join(os.getcwd(), config["download_data_from_ZFIN"]["dir"], "xpat_stage_anatomy.tsv"),
        xpat_fish = os.path.join(os.getcwd(), config["download_data_from_ZFIN"]["dir"], "xpat_fish.tsv"),
        anatomy_synonyms = os.path.join(os.getcwd(), config["download_data_from_ZFIN"]["dir"], "anatomy_synonyms.tsv")
    shell:
        "wget -O {output.anatomy_item} {input.anatomy_item};" \
        "wget -O {output.anatomy_synonyms} {input.anatomy_synonyms};" \
        "wget -O {output.xpat_stage_anatomy} {input.xpat_stage_anatomy};" \
        "wget -O {output.xpat_fish} {input.xpat_fish};"

And this is the content of my configs.yaml file:

ZFIN_url:
  # Zebrafish Anatomy Term
  anatomy_item: "https://zfin.org/downloads/file/anatomy_item.txt"
  # Zebrafish Gene Expression by Stage and Anatomy Term
  xpat_stage_anatomy: "https://zfin.org/downloads/file/xpat_stage_anatomy.txt"
  # ZFIN Genes with Expression Assay Records
  xpat_fish: "https://zfin.org/downloads/file/xpat_fish.txt"
  # Zebrafish Anatomy Term Synonyms
  anatomy_synonyms: "https://zfin.org/downloads/file/anatomy_synonyms.txt"

download_data_from_ZFIN:
  dir: ZFIN_data

The error message is:

Building DAG of jobs...
MissingInputException in line 10 of /home/zhangdong/works/NGS/coevolution/snakemake/coevolution.rule:
Missing input files for rule download_data_from_ZFIN:
https://zfin.org/downloads/file/anatomy_item.txt

I want to make sure that if this exception is caused by none file input for the input rule?

Take a look here: https://stackoverflow.com/questions/59053035/snakemake-rule-that-downloads-data. The file requires no input, so listing urls are required files on your computer is wrong. You could put those urls as params instead of input — Maarten-vd-Sande, Feb 05 '21 at 17:41
Hi Maarten-vd-Sande, thank you very much, as you suggested, it worked when I remove the `input` part. — Dong Zhang, Feb 06 '21 at 01:46

score 1 · Answer 1 · answered Feb 06 '21 at 10:48

Note that you can also use remote files as input so you may avoid rule download_data_from_ZFIN altogether. E.g.:

from snakemake.remote.HTTP import RemoteProvider as HTTPRemoteProvider

HTTP = HTTPRemoteProvider()

rule all:
    input:
        'output.txt',

rule one:
    input:
        # Some file from the web
        x= HTTP.remote('https://plasmodb.org/common/downloads/release-49/PbergheiANKA/txt/PlasmoDB-49_PbergheiANKA_CodonUsage.txt', keep_local=True)
    output:
        'output.txt',
    shell:
        r"""
        # Do something with the remote file
        head {input.x} > {output}
        """

The remote file will be downloaded and stored locally under plasmodb.org/common/.../PlasmoDB-49_PbergheiANKA_CodonUsage.txt

score 0 · Answer 2 · answered Feb 08 '21 at 01:26

Many thanks @dariober, I tried the follwing code and it worked,

import os
from snakemake.remote.HTTP import RemoteProvider as HTTPRemoteProvider
configfile: "Configs.yaml"

HTTP = HTTPRemoteProvider()

rule all:
    input:
        expand(os.path.join(os.getcwd(),config["download_data_from_ZFIN"]["dir"],"{item}.tsv"),
            item=list(config["ZFIN_url"].keys()))

rule download_data_from_ZFIN:
    input:
        lambda wildcards: HTTP.remote(config["ZFIN_url"][wildcards.item], keep_local=True)
    output:
        os.path.join(os.getcwd(),config["download_data_from_ZFIN"]["dir"],"{item}.tsv")
    threads:
        1
    shell:
        "mv {input} > {output}"

Such code is more snakemake-like, but I have two further questions:

Is there a way to specify the output file name for the downloading? Now I use the mv command to achieve that.
Does this remote files function support parallel works? I tried the above code together with --cores 6, but it still download the file one by one.

Does snakemake support none file Input?

2 Answers2