AzureML SDK problems with pip-installed dependencies in environment file

Question

i am try to run a Python Script in AzureML SDK. However in the runs log file, it prints the same error over and over again:

Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies. Conda may not use the correct pip to install your packages, and they may end up in the wrong place. Please add an explicit pip dependency. I'm adding one for you, but still nagging you.

The defined conda environment looks like this:

    {
    "databricks": {
        "eggLibraries": [],
        "jarLibraries": [],
        "mavenLibraries": [],
        "pypiLibraries": [],
        "rcranLibraries": []
    },
    "docker": {
        "arguments": [],
        "baseDockerfile": null,
        "baseImage": "mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04:20210301.v1",
        "baseImageRegistry": {
            "address": null,
            "password": null,
            "registryIdentity": null,
            "username": null
        },
        "enabled": false,
        "platform": {
            "architecture": "amd64",
            "os": "Linux"
        },
        "sharedVolumes": true,
        "shmSize": null
    },
    "environmentVariables": {
        "EXAMPLE_ENV_VAR": "EXAMPLE_VALUE"
    },
    "inferencingStackVersion": null,
    "name": "MyEnvironment04",
    "python": {
        "baseCondaEnvironment": null,
        "condaDependencies": {
            "channels": [
                "anaconda",
                "conda-forge"
            ],
            "dependencies": [
                "python=3.6.2",
                {
                    "pip": [
                        "pip=20.2.40"
                    ]
                },
                "scikit-learn"
            ],
            "name": "azureml_815589d460c271a1415198e7283fa9e9"
        },
        "condaDependenciesFile": null,
        "interpreterPath": "python",
        "userManagedDependencies": false
    },
    "r": null,
    "spark": {
        "packages": [],
        "precachePackages": true,
        "repositories": []
    },
    "version": "1"
}

I am assuming that the definition for my pip package in my environment is wrong.

# Create the dependencies object
myenv_dep = CondaDependencies.create(conda_packages=['scikit-learn'], pip_packages=['pip=20.2.40'])
myenv.python.conda_dependencies = myenv_dep

Please let me know if i need to provide further information.

Thank you!

Subhrajeit Bhowmick · Answer 1 · 2021-08-11T13:41:32.333

You might consider deleting your azureml_conda_lock directory if any and try again. The location of this directory is OS-dependent but you should find it under .azureml/locks/ location.

Else I am assuming you have your AzureML SDK set up within a Python environment like VENV or similar non-Conda environment.

That is why AzureML SDK is getting into a loop of failed attempts to set up a Conda environment from within your VENV / non-Conda environment, as a result of which it is logging such message.

In such case, you should try setting up your AzureML SDK using Conda and that should fix the issue.

AzureML SDK problems with pip-installed dependencies in environment file

1 Answers1