Snakemake: Is it possible to use directories as wildcards?

Question

Hi I´m new in Snakemake and have a question. I want to run a tool to multiple data sets. One data set represents one tissue and for each tissue exists fastq files, which are stored in the respective tissue directory. The rough command for the tools is:

  python TEcount.py -rosette rosettefile -TE te_references -count result/tissue/output.csv -RNA <LIST OF FASTQ FILE FOR THE RESPECTIVE SAMPLE>

The tissues shall be the wildcards. How can I do this? Below I have a first try that did not work.

import os                                                                        

#collect data sets                                                               
SAMPLES=os.listdir("data/rnaseq/")                                               


rule all:                                                                        
    input:                                                                       
        expand("results/{sample}/TEtools.{sample}.output.csv", sample=SAMPLES)                   

rule run_TEtools:                                                                
    input:                                                                       
        TEcount='scripts/TEtools/TEcount.py',                                    
        rosette='data/prepared_data/rosette/rosette',                            
        te_references='data/prepared_data/references/all_TE_instances.fa'        
    params:
        #collect the fastq files in the tissue directory                                                              
        fastq_files = os.listdir("data/rnaseq/{sample}")                         
    output:                                                                      
        'results/{sample}/TEtools.{sample}.output.csv'                           
    shell:                                                                       
        'python {input.TEcount} -rosette {input.rosette} -TE                     
{input.te_references} -count {output} -RNA {params.fastq_files}'

In the rule run_TEtools it does not know what the {sample} is.

score 2 · Accepted Answer · answered Jul 08 '19 at 12:29

A snakemake wildcard can be anything. It is basically just a string.
There are some issues with the way you are trying to achieve what you want.

Ok, here's how I would do it. Explanations follow:

import os                                                                        

#collect data sets
# Beware no other directories or files (than those containing fastqs) should be in that folder                                                        
SAMPLES=os.listdir("data/rnaseq/")                                               

def getFastqFilesForTissu(wildcards):
    fastqs = list()
    # Beware no other files than fastqs should be there
    for s in os.listdir("data/rnaseq/"+wildcards.sample):
        fastqs.append(os.path.join("data/rnaseq",wildcards.sample,s))
    return fastqs

rule all:                                                                        
    input:                                                                       
        expand("results/{sample}/TEtools.{sample}.output.csv", sample=SAMPLES)                   

rule run_TEtools:                                                                
    input:                                                                       
        TEcount='scripts/TEtools/TEcount.py',                                    
        rosette='data/prepared_data/rosette/rosette',                            
        te_references='data/prepared_data/references/all_TE_instances.fa',
        fastq_files = getFastqFilesForTissu        
    output:                                                                      
        'results/{sample}/TEtools.{sample}.output.csv'                           
    shell:                                                                       
        'python {input.TEcount} -rosette {input.rosette} -TE {input.te_references} -count {output} -RNA {input.fastq_files}'

First of all, your fastq file should be defined as inputs in order for snakemake to know that they are files and that if they are changed, rules must be rerun. It is quite bad practice to define input files as params. params are made for parameters, usually not for files.
Second, your script file is defined as input. You have to be aware that everytime you modify it, rules will be rerun. Maybe that's what you want.

I would use a defined function to get the fastq file in each directory. If you want to use a function (like os.listdir()), you can't use your wildcards directly. You have to inject it in the function as a python object. You can either define a function that will take one argument, a wildcard object containing all your wildcards, or use the lambda keyword (ex: input = lamdba wildcards: myFuntion(wildcards.sample)).
Another problem you have is that os.listdir() returns a list of files without the relative path. Also beware that the order in which os.listdir() will return you fastq file is unknown. Maybe that doesn't matter for your command.

Nice, thank you very much! It works. One more silly question, sorry, but why can I call the getFastqFilesForTissu function without brackets? Is it a snakemake thing? — Rob, Jul 08 '19 at 13:09
Hi Rob, Regarding the use of `getFastqFilesForTissu` rather than `getFastqFilesForTissu()`. It is not a snakemake thing; it is accepted python syntax. In the first case you are storing the function (i.e. the block of code). In the second (with parentheses), you are storing the results of the function (whatever is returned). Snakemake must be written in such a way that if a function definition is provided as input, then snakemake will execute that block of code to generate the expected list of input files. — Tim, Jul 08 '19 at 13:56
@Rob yes, in fact you cannot use `getFastqFilesForTissu()` because snakemake will throw a "missing 1 required positional argument" exception (for the same reason you cannot use the wildcards directly when using a function). You can, however, do `fastq_files = lambda wildcards: getFastqFilesForTissu(wildcards)`. My understanding of python is too poor to explain it simply... — Eric C., Jul 08 '19 at 14:21

Snakemake: Is it possible to use directories as wildcards?

1 Answers1