Why is env different in an Azure ML notbook and an Azure ML terminal?

Question

I'm using MS Azure ML and have found that when I start a Notebook (from the Azure ML Studio) it is executing in a a different environment than if I create a Python script and run it from the studio. I want to be able to create a specific environment and have the Notebook use that. The environment that the Notebook seems to run does not contain the packages I need and I want to preserve different environments.

Answer 1

First open a terminal, using the same compute target as you want to use with your Notebook afterwards, and to use and existing environment you can do:

conda activate existing_env
conda install ipykernel
python -m ipykernel install --user --name existing_env --display-name "Python 3.8 - Existing Environment"   

However, to create a new environment and use it in you AzureML Notebook, you have to do the following commands:

conda create --name new_env python=3.8
conda activate new_env
conda install pip
conda install ipykernel
python -m ipykernel install --user --name new_env --display-name "Python 3.8 - New Environment"

And then last, but not least, you have to edit the Jupyter Kernel display names:

IMPORTANT Please ensure you are comfortable running all these steps:

jupyter kernelspec list
cd <folder-that-matches-the-kernel-of-your-environment>
sudo nano kernel.json

Then edit the name to match what you want and save the file.

Answer 2

Using Pip, Conda, Docker image & Dockerfile we can create ML environments for your own scripts.

I want to be able to create a specific environment

To manually create an environment,

from azureml.core.environment import Environment
Environment(name="myenv")

Use Conda dependencies or pip requirement files for creating an environment using below code:

#### From a Conda specification file
myenv = Environment.from_conda_specification(name = "myenv",
                                             file_path = "path-to-conda-specification-file")

#### From a pip requirements file
myenv = Environment.from_pip_requirements(name = "myenv",
                                          file_path = "path-to-pip-requirements-file")

The environment that the Notebook seems to run does not contain the packages I need and I want to preserve different environments.

To add the packages to your environment, can use the Conda, pip or private wheel files but it is recommended to use CondaDependency class.

from azureml.core.environment import Environment
from azureml.core.conda_dependencies import CondaDependencies

myenv = Environment(name="myenv")
conda_dep = CondaDependencies()

#### Installs numpy version 1.17.0 conda package
conda_dep.add_conda_package("numpy==1.17.0")

#### Installs pillow package
conda_dep.add_pip_package("pillow")

#### Adds dependencies to PythonSection of myenv
myenv.python.conda_dependencies=conda_dep

Register your environment in workspace using the command myenv.register(workspace=ws)

list(workspace) the list the environments in the workspace

To run your specific environment:

from azureml.core import Run
Run.get_environment()

To install a Conda environment as a kernel in a notebook, see add a new Jupyter kernel and the code should be followed in a same sequence:

conda create --name newenv
conda activate newenv
conda install pip
conda install ipykernel
python -m ipykernel install --user --name newenv --display-name "Python (newenv)"

To add, change or remove the Jupyter Kernels, please refer this article.

Refer this Document for more details