5

If we have PyPi Packages added as Artifacts to an Azure DevOps Project Feed, how can we use these packages as a source for installing packages in DatabricksStep of Azure Machine Learning Service?

While using pip in any environment, we use our Azure DevOps Project Artifacts feed in the following way:

pip install example-package --index-url=https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/

The DatabricksStep class of the Azure Machine Learning Service accepts the following parameters:

python_script_name = "<Some-Script>.py"
source_directory = "<Path-To-Script>"

<Some-Placeholder-Name-for-the-step> = DatabricksStep(
    name=<Some-Placeholder-Name-for-the-step>,
    num_workers=1,
    python_script_name=python_script_name,
    source_directory=source_directory,
    run_name= <Name-of-the-run>,
    compute_target=databricks_compute,
    pypi_libraries = [
                      PyPiLibrary(package = 'scikit-learn'), 
                      PyPiLibrary(package = 'scipy'), 
                      PyPiLibrary(package = 'azureml-sdk'), 
                      PyPiLibrary(package = 'joblib'), 
                      PyPiLibrary(package = 'azureml-dataprep[pandas]'),
                      PyPiLibrary(package = 'example-package', repo='https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/')
                    ], 

    allow_reuse=True
)

However, PyPiLibrary(package = 'example-package', repo='https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/') will give an error. How exactly should we consume the Artifacts Feed as an input to the PyPiLibrary property of the DatabricksStep Class in Azure Machine Learning Service?

Anirban Saha
  • 1,350
  • 2
  • 10
  • 38

0 Answers0