0

I am trying to run a Machine Learning training script I wrote in azure by using:

env = Environment.from_conda_specification("experiment_env", "my_environment.yml")
script_config = ScriptRunConfig(source_directory=experiment_folder,
                                script='path_to_file/classifier_train.py',
                                arguments=arguments,
                                environment=env,
                                docker_runtime_config=DockerConfiguration(use_docker=True)

my problem is, that in order to run the code of classifier_train.py I need to call

pip install -e .

since I need code from a local package I wrote (the setup.py for this package is in the experiment folder).

Can anyone tell me how I can run pip install -e . after azure installed all other package when building the environment and before running the classifier_train.py script?

mayool
  • 138
  • 8

1 Answers1

1

To install local libraries on azure ML environment, we need to create the Data Science VM on the local machine and connect it with the workspace subscription details.

Create ML studio workspace and download the JSON file which consists of the details of the workspace need to be connected to the VM.

  1. Create a python virtual environment on the local machine
  2. Either virtualenv or conda is fine
  3. Activate the virtual environment
  4. Install the Azure Machine Learning Python SDK
  5. Configure the Azure ML Studio with local machine

enter image description here

  1. Open the Jupyter Notebook installed and configured. Create a virtual environment.
  2. conda install notebook ipykernel – enable all the ipykernal things
  3. ipython kernel install --user --name <myenv> --display-name "Python (myenv)" – creating a kernel
  4. Launch the jupyter notebook server

Now we need to get the ARM template by following the procedure taken from MS Docs

Using the following code block create Windows DSVM

az vm create --resource-group YOUR-RESOURCE-GROUP-NAME --name YOUR-VM-NAME --image microsoft-dsvm:dsvm-windows:server-2016:latest --admin-username YOUR-USERNAME --admin-password YOUR-PASSWORD --authentication-type password

Using the following code block create Ubuntu DSVM

az vm create --resource-group YOUR-RESOURCE-GROUP-NAME --name YOUR-VM-NAME --image microsoft-dsvm:linux-data-science-vm-ubuntu:linuxdsvmubuntu:latest --admin-username YOUR-USERNAME --admin-password YOUR-PASSWORD --generate-ssh-keys --authentication-type password

Create conda environment:

conda create -n py310 python=310

Activate the environment and install the libraries locally which are directly impacted in azure ml platform

conda activate py310
pip install azure-ai-ml
Sairam Tadepalli
  • 1,563
  • 1
  • 3
  • 11
  • Thanks for the awesome answer! I am sure your solution would work... I solved the problem myself a couple of days ago by adding: - pip: - -e git+https://github.com/myrepo#egg=myenvname to the environment.yml file – mayool Oct 27 '22 at 12:23