2

If we have an AzureML Pipeline published, how can we trigger it from Azure DevOps without using Python Script Step or Azure CLI Step?

The AzureML Steps supported natively in Azure DevOps include Model_Deployment and Model_Profiling.

Is there any step in Azure DevOps which can be used to directly trigger a published Azure Machine Learning Pipeline while maintaining capabilities like using Service Connections and passing environmental variables, Gated Release (Deployment)?

Edit: This process can then be used to run as an agentless job.

Anirban Saha
  • 1,350
  • 2
  • 10
  • 38

2 Answers2

3

I am afraid there is no other steps available in Azure Devops which can directly trigger a published azure ml pipeline. You have to use Python Script Step or Azure CLI Step in azure devops pipeline to trigger azure ml pipeline.

To trigger azure ml pipeline using azure cli task in azure devops pipeline. You can check out below steps.

1, Create an azure pipeline. See example here.

2, Create an azure Resource Manager service connection to connect your Azure subscription to Azure devops. See this thread for an example

3, Add Az cli task in your yaml pipeline. Run below scripts as inline scripts. See document here for more information.

steps:
- task: AzureCLI@2
  displayName: 'Azure CLI '
  inputs:
    azureSubscription: 'azure Resource Manager service connection'
    scriptType: ps
    scriptLocation: inlineScript
    inlineScript: |
     
     #To install the Machine Learning CLI extension
     az extension add -n azure-cli-ml

     az ml run submit-pipeline --pipeline-id "{id}"

Update:

If you want to avoid using build agents. You can run the invoke rest api task in an agentless job. See below steps:

1, Create a Generic service connection in azure devops. See here for creating service connection.

2, Add below url as the Server URL of the generic service connection. See here for more information about below url.

enter image description here

3, Add a agentless job(server job) in your pipeline. Add invoke rest api task in this agentless job. So that, the pipeline will execute the invoke rest api task to trigger the azureml pipeline without using a build agent.

You can also setup an azure logic app in your azure subscription.

You can set the logic app trigger as azure devops events. Or you can set a http request as the trigger events(You can the use invoke rest api task or azure devops web hook to call this http request to trigger this logic app).

And then add a HTTP action with the url as above url screenshot. Please see here for more information.

Levi Lu-MSFT
  • 27,483
  • 2
  • 31
  • 43
  • Yes, I am aware of this option and this is how I had been triggering the AML Pipelines from ADO. However, due to some business-specific use case, I need to bypass the need to set up a build agent which this `az-CLI` task will need to run. Is there any other way? Can this be a feature request to MSFT? – Anirban Saha Feb 02 '21 at 04:51
  • @AnirbanSaha You can have a try using the api to trigger the azure ml from azure devops. See above update. – Levi Lu-MSFT Feb 02 '21 at 06:40
  • Can the Azure Resource Manager Service Connection scoped at the Resource Group level be used to trigger the pipeline as Invoke Rest API? https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-pipelines#run-a-published-pipeline – Anirban Saha Feb 02 '21 at 07:21
  • @AnirbanSaha I am afraid it cannot. – Levi Lu-MSFT Feb 02 '21 at 08:22
1

Assumptions:

  1. An AzureML Pipeline is published and the REST endpoint is ready- To be referred to in this answer as <AML_PIPELINE_REST_URI>. And Published Pipeline ID is also ready- To be referred to in this answer as <AML_PIPELINE_ID>
  2. You have the Azure Machine Learning Extension installed: Azure Machine Learning Extension

To Invoke the Azure Machine Learning Pipeline we use the Invoke ML Pipeline step available in Azure DevOps. It is available when running an Agentless Job.

To trigger it the workflow is as follows:

  1. Create a New Pipeline. Using the Classic Editor, delete the default Agent Job 1 stage. enter image description here

enter image description here

  1. Add an agentless job: enter image description here

  2. Add a task to this Agentless Job: enter image description here

  3. Use AzureML Published Pipeline Task: enter image description here

  4. Use the Service Connection Mapped to the AML Workspace. You can find more on this at the official documentation enter image description here

  5. Choose the Pipeline to trigger using the <AML_PIPELINE_ID>: enter image description here

  6. Give The experiment name and Pipeline Parameters if any: enter image description here

  7. That's it, you can Save and Queue: enter image description here

Alternatively, you can simply use the following jobs:

- job: Job_2
  displayName: Agentless job
  pool: server
  steps:
  - task: MLPublishedPipelineRestAPITask@0
    displayName: Invoke ML pipeline
    inputs:
      connectedServiceName: <REDACTED-AML-WS-Level-Service_Connection-ID>
      PipelineId: <AML_PIPELINE_ID>
      ExperimentName: experimentname
      PipelineParameters: ''
Anirban Saha
  • 1,350
  • 2
  • 10
  • 38