0

I have a zeppelin running in an EMR cluster. This zeppelin has multiple users who login into the zeppelin via a Shiro based authentication method. I need a way to create virtual environments for each user so that they can manage their own pip dependencies. I do not want to install pip libraries for all the users globally. Instead, I need a way for different users to use different library versions inside the same zeppelin environment. Is this possible in the Zeppelin and if so, how?

Nikhil Mudholkar
  • 33
  • 8

2 Answers2

1

Setting the "interpreter binding modes" to "isolated" (or even "scoped") might solve your problem.

Isolated mode runs a separate interpreter process for each note in the case of per note scope. So, each note has an absolutely isolated session.

https://zeppelin.apache.org/docs/0.10.0/usage/interpreter/interpreter_binding_mode.html

It's not a python virtual environment per se, but maybe the provided isolation could work.

MEMark
  • 1,493
  • 2
  • 22
  • 32
0

Answering my own question here as I have figured it out.

This can be done by configuring anaconda inside the zeppelin and running the interpreter in isolated mode per user. Zeppelin has inbuilt support for anaconda, its just that one needs to do some configurations to make it work.

These configuration steps are:-

  1. Create a symlink to /bin/conda: ln -s /opt/anaconda3/bin/conda /bin/conda
  2. Create a symlink to /bin/python: ln -s /opt/anaconda3/bin/python /bin/python
  3. In the settings for the Python interpreter, set zeppelin.python to /opt/anaconda3/bin/python3
  4. Add python path to ~/.bashrc export PATH=/home/hadoop/miniconda/bin:$PATH

If you are spinning up zeppelin using terraform, you can use this shell script to setup anaconda as a part of bootstrap actions,

    #!/bin/bash

sudo wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
sudo bash ~/miniconda.sh -b -p $HOME/miniconda
echo "export PATH=/home/hadoop/miniconda/bin:$$PATH" >> ~/.bashrc
sudo ln -s /home/hadoop/miniconda/bin/conda /usr/bin/conda
sudo conda install -y python=3.7 anaconda=custom
Nikhil Mudholkar
  • 33
  • 8