I am currently experimenting executing my snakemake workflow no GKE. I have noticed huge time discrepancies in the time it takes for the same conda environment to be setup on different pods.
For example, on this pod the env fastq2bam took almost 40 minutes to download and install.
2021-10-04T21:30:12.013016030ZBuilding DAG of jobs...
2021-10-04T21:30:14.088986292ZCreating conda environment workflow/envs/fastq2bam.yml...
2021-10-04T21:30:14.089894547ZDownloading and installing remote packages.
2021-10-04T22:09:06.496554997ZEnvironment for workflow/envs/fastq2bam.yml created (location: .snakemake/conda/a6dd1d801207b0464f4fb8c9ad01dfa5)
While, on a different pod it only took about 3 minutes to build the same env.
2021-10-04T21:32:41.695487553ZBuilding DAG of jobs...
2021-10-04T21:32:43.602866802ZCreating conda environment workflow/envs/fastq2bam.yml...
2021-10-04T21:32:43.603565305ZDownloading and installing remote packages.
2021-10-04T21:35:03.559349719ZEnvironment for workflow/envs/fastq2bam.yml created (location: .snakemake/conda/a6dd1d801207b0464f4fb8c9ad01dfa5)
The only real difference I can think of between the pods is the amount of CPUs being requested by them. The slow pod is requesting 1 core, while the quicker one has 10 cores. But I don't think that CPUs could be the issue here?
