Azure ML not able to create conda environment (exit code: -15)

Question

When I try to run the experiment defined in this notebook in notebook, I encountered an error when it is creating the conda env. The error occurs when the below cell is executed:

from azureml.core import Experiment, ScriptRunConfig, Environment
from azureml.core.conda_dependencies import CondaDependencies
from azureml.widgets import RunDetails


# Create a Python environment for the experiment
sklearn_env = Environment("sklearn-env")

# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip'],
                                    pip_packages=['azureml-defaults','azureml-dataprep[pandas]'])
sklearn_env.python.conda_dependencies = packages

# Get the training dataset
diabetes_ds = ws.datasets.get("diabetes dataset")

# Create a script config
script_config = ScriptRunConfig(source_directory=experiment_folder,
                              script='diabetes_training.py',
                              arguments = ['--regularization', 0.1, # Regularizaton rate parameter
                                           '--input-data', diabetes_ds.as_named_input('training_data')], # Reference to dataset
                              environment=sklearn_env)

# submit the experiment
experiment_name = 'mslearn-train-diabetes'
experiment = Experiment(workspace=ws, name=experiment_name)
run = experiment.submit(config=script_config)
RunDetails(run).show()
run.wait_for_completion() 

Everytime I run this, I always faced the issue of creating the conda env as below:

Creating conda environment...
Running: ['conda', 'env', 'create', '-p', '/home/azureuser/.azureml/envs/azureml_000000000000', '-f', 'azureml-environment-setup/mutated_conda_dependencies.yml']
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

Installing pip dependencies: ...working... 

Attempting to clean up partially built conda environment: /home/azureuser/.azureml/envs/azureml_000000000000
Remove all packages in environment /home/azureuser/.azureml/envs/azureml_000000000000:
Creating conda environment failed with exit code: -15

I could not find anything useful on the internet and this is not the only script where it fail. When I am try to run other experiments I have sometimes faced this issue. One solution which worked in the above case is I moved the pandas from pip to conda and it was able to create the coonda env. Example below:

# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip'],
                                    pip_packages=['azureml-defaults','azureml-dataprep[pandas]'])

# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip','pandas'],
                                    pip_packages=['azureml-defaults','azureml-dataprep'])

The error message (or the logs from Azure) is also not much help. Would apprecite if a proper solution is available.

Edit: I have recently started learning to use Azure for Machine learning and so if I am not sure if I am missing something? I assume the example notebooks should work as is hence raised this question.

Answer 1

short answer

Totally been in your shoes before. This code sample seems a smidge out of date. Using this notebook as a reference, can you try the following?

packages = CondaDependencies.create(
    pip_packages=['azureml-defaults','scikit-learn']
)

longer answer

Using pip with Conda is not always smooth sailing. In this instance, conda isn't reporting up the issue that pip is having. The solution is to create and test this environment locally where we can get more information, which will at least will give you a more informative error message.

1. Install anaconda or miniconda (or use an Azure ML Compute Instance which has conda pre-installed)
2. Make a file called environment.yml that looks like this

name: aml_env
dependencies:
 - python=3.8
 - pip=21.0.1
 - pip:
    - azureml-defaults
    - azureml-dataprep[pandas]
    - scikit-learn==0.24.1

3. Create this environment with the command conda env create -f environment.yml.
4. respond to any discovered error message
5. If there' no error, use this new environment.yml with Azure ML like so

sklearn_env = Environment.from_conda_specification(name = 'sklearn-env', file_path = './environment.yml')

more context

the error I'm guessing that's happening is when you reference a pip requirements file from a conda environment file. In this scenario, conda calls pip install -r requirements.txt and if that command errors out, conda can't report the error.

requirements.txt

scikit-learn==0.24.1
azureml-dataprep[pandas]

environment.yml

name: aml_env
dependencies:
 - python=3.8
 - pip=21.0.1
 - pip:
    - -rrequirements.txt

Answer 2

What worked for me looking at the previous notebook 05 - Train Models.ipynb:

packages = CondaDependencies.create(conda_packages=['pip', 'scikit-learn'],
                                    pip_packages=['azureml-defaults'])

You have to:

1. Remove 'azureml-dataprep[pandas]' from pip_packages
2. Change the order of conda_packages - pip should go first