4

I am using snakemake for a long and complicated pipeline which involves some externally written python2 scripts. When I try to specify python2 with pyenv, pyenv shell command fails while pyenv global and pyenv local have no effect. I have two questions.

  • Why is the shell command absent? Is it because snakemake runs a non-login non-interactive shell?
  • Is it possible at all to switch to python2 in a snakemake rule using pyenv?

Here is an example Snakefile.

rule aaa:
   output:
      "aaa.txt"
   shell:
      """
      pyenv versions
      python --version
      echo "global"
      pyenv global 2.7.12
      python --version
      echo "local"
      pyenv local 2.7.12
      python --version
      echo "shell"
      pyenv shell 2.7.12
      python --version
      echo "pa-pa, zlyj svite" > aaa.txt
      """

That produces the following output.

...
rule aaa:
    output: aaa.txt
    jobid: 0

  system
  2.7.12
  3.5.2
* 3.6.1 (set by PYENV_VERSION environment variable)
Python 3.6.1
global
Python 3.6.1
local
Python 3.6.1
shell
pyenv: no such command `shell'
Error in job aaa while creating output file aaa.txt.
...
rbrisk
  • 306
  • 1
  • 7
  • 1
    Hi, I don't use pyenv. So I can't help you to know why it doesn't recognize it. But Snakemake recognizes conda to set environment. So you can use it just like in the following exemple here : http://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html?highlight=conda . Another solution that works it is to deal with module to set environment inside a shell rule. – Pereira Hugo Jun 02 '17 at 10:13
  • Thank you for your suggestions. I ended up dropping pyenv entirely. I installed python2 and python3 as lmod modules and that worked. – rbrisk Jun 02 '17 at 13:02
  • 1
    I can only recommend to try integrated package management via conda: http://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html?highlight=conda#integrated-package-management – Johannes Köster Jun 03 '17 at 11:12

1 Answers1

1

In case someone else needs this functionality, I have a partial answer. As far as I can tell, when snakemake spawns a shell to run a command, the shell does not read $HOME/.bashrc, $HOME/.bash_profile, or $HOME/.profile. I tested it by adding an environmental variable to one of those files and then trying to read its value in a Snakefile using an existing shell (i.e. shell that does not have that variable set). However, the spawned shell inherits the current shell's environment. That is equivalent to adding $PYENV_ROOT/bin to your $PATH without running pyenv init. As a consequence, pyenv command is there for the shell to find but it is not fully operational.

So, to use pyenv in Snakefile, you just need to add eval "$(pyenv init -)" at the beginning of each rule where you want to use it. For example, the following Snakefile works as intended.

rule aaa:
   output:
      "aaa.txt"
   shell:
      """
      eval "$(pyenv init -)"
      pyenv versions
      python --version
      echo "shell"
      pyenv shell 2.7.12
      python --version
      echo "pa-pa, zlyj svite" > aaa.txt
      """

producing the following output

...
  system
  2.7.12
  3.5.2
* 3.6.1 (set by PYENV_VERSION environment variable)
Python 3.6.1
shell
Python 2.7.12
...

I still don't understand why none of the rc and profile files are read.

rbrisk
  • 306
  • 1
  • 7