I have a pipeline which works just fine in the command line
snakemake -l --snakefile snakemake_example/sankefile_test9.txt
I would like to be able to use it on the cluster. The pipeline takes samples (specified in the config file), and runs a few steps of processing - this is an RNA-Seq pipeline. I've tried to submit to the cluster using these two different ways
First try
snakemake --snakefile sankefile_test9_config.txt --jobs 999 --cluster 'bsub -q bio -R "rusage[mem=4000]"'
Second try
snakemake --snakefile sankefile_test9_config.txt --cluster 'bsub -q bio' -j
which yielded the following output
Provided cluster nodes: 48
Job counts:
count jobs
1 all
2 collate_barcodes
2 correct_counts
2 count_reads
2 dedup_counts
2 extract_gz_samples
2 mark_duplicaticates
2 move_bc
2 run_cutadapt
2 star_mapping
19
rule extract_gz_samples:
input: cluster_fastq/Zelzer_M_Spindle_M_1.R1.fastq.gz, cluster_fastq/Zelzer_M_Spindle_M_1.R2.fastq.gz
output: cluster_fastq/Zelzer_M_Spindle_M_1.R1.fastq, cluster_fastq/Zelzer_M_Spindle_M_1.R2.fastq
wildcards: sample=cluster_fastq/Zelzer_M_Spindle_M_1
Memory reservation is (MB): 2048
Memory Limit is (MB): 2048
rule extract_gz_samples:
input: cluster_fastq/WT_M_DT_T_393.R1.fastq.gz, cluster_fastq/WT_M_DT_T_393.R2.fastq.gz
output: cluster_fastq/WT_M_DT_T_393.R1.fastq, cluster_fastq/WT_M_DT_T_393.R2.fastq
wildcards: sample=cluster_fastq/WT_M_DT_T_393
Memory reservation is (MB): 2048
Memory Limit is (MB): 2048
Waiting at most 5 seconds for missing files.
Exception in thread Thread-1:
Traceback (most recent call last):
File "/apps/RH6U4/python/3.5.2/lib/python3.5/site-packages/snakemake/dag.py", line 257, in check_and_touch_output
wait_for_files(expanded_output, latency_wait=wait)
File "/apps/RH6U4/python/3.5.2/lib/python3.5/site-packages/snakemake/io.py", line 341, in wait_for_files
latency_wait, "\n".join(get_missing())))
OSError: Missing files after 5 seconds:
cluster_fastq/Zelzer_M_Spindle_M_1.R1.fastq
cluster_fastq/Zelzer_M_Spindle_M_1.R2.fastq
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/apps/RH6U4/python/3.5.2/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/apps/RH6U4/python/3.5.2/lib/python3.5/threading.py", line 862, in run self._target(*self._args, **self._kwargs)
File "/apps/RH6U4/python/3.5.2/lib/python3.5/site-packages/snakemake/executors.py", line 517, in _wait_for_jobs self.finish_job(active_job.job)
File "/apps/RH6U4/python/3.5.2/lib/python3.5/site-packages/snakemake/executors.py", line 426, in finish_job
super().finish_job(job, upload_remote=False) File "/apps/RH6U4/python/3.5.2/lib/python3.5/site-packages/snakemake/executors.py", line 153, in finish_job
super().finish_job(job, upload_remote=upload_remote) File "/apps/RH6U4/python/3.5.2/lib/python3.5/site-packages/snakemake/executors.py", line 111, in finish_job
self.dag.check_and_touch_output(job, wait=self.latency_wait)
File "/apps/RH6U4/python/3.5.2/lib/python3.5/site-packages/snakemake/dag.py", line 259, in check_and_touch_output
raise MissingOutputException(str(e), rule=job.rule)
snakemake.exceptions.MissingOutputException: Missing files after 5 seconds:
cluster_fastq/Zelzer_M_Spindle_M_1.R1.fastq
cluster_fastq/Zelzer_M_Spindle_M_1.R2.fastq^C^H^CTerminating processes on user request.
Will exit after finishing currently running jobs.
Removing output files of failed job extract_gz_samples since they might be corrupted:
cluster_fastq/Zelzer_M_Spindle_M_1.R1.fastq, cluster_fastq/Zelzer_M_Spindle_M_1.R2.fastq
Removing output files of failed job extract_gz_samples since they might be corrupted:
cluster_fastq/WT_M_DT_T_393.R1.fastq, cluster_fastq/WT_M_DT_T_393.R2.fastq
At this point, the program seems to be stuck (see the ^C^H^C, my break) - and if I check the jobs (in another session) using bjobs, there are no jobs on the queue.
Any idea why this is happening and how to debug it?
