0

I have an Azure Pipeline Build. The *.yaml file executes correctly a Python script (PythonScript@0). This script itself creates (if does not exist), executes and publishes Azure ML pipeline. It runs well when the Build is executed manually or is triggered by commits.

But I want to schedule the automated execution of the ML pipeline (Python script) on a daily basis.

I tried the following approach:

pipeline_id = published_pipeline.id
recurrence = ScheduleRecurrence(frequency="Day", interval=1)
recurring_schedule = Schedule.create(ws, 
                                     name=<schedule_name>, 
                                     description="Title", 
                                     pipeline_id=pipeline_id,  
                                     experiment_name=<experiment_name>, 
                                     recurrence=recurrence)

In this case the pipeline runs during 3-4 seconds and terminates successfully. However, the Python script is not executed.

Also, I tried to schedule the execution of a pipeline using Build, but I assume that it is a wrong approach. It rebuilds a pipeline, but I need to execute a previously published pipeline.

schedules:
- cron: "0 0 * * *"
  displayName: Daily build
  always: true

How can I execute my published pipeline daily? Should I use Release (which agents, which tasks?)?

Fluxy
  • 2,838
  • 6
  • 34
  • 63
  • `Also, I tried to schedule the execution of a pipeline using Build, but I assume that it is a wrong approach. It rebuilds a pipeline, but I need to execute a previously published pipeline.` Hi, what's the steps and tasks in your build-pipeline? If you're trying to schedule the daily execution of python script, you can choose to schedule the execution of the build pipeline or create a new pipeline to execute the python script. – LoLance Dec 09 '19 at 02:41
  • When you said python script is not executed? Did you mean PythonScript task is not excuted? Could you check if the ML pipeline is created and published to azure correctly. And you can try publishing the scheduled ML pipeline manually to azure to check if the schedule is working. – Levi Lu-MSFT Dec 27 '19 at 07:56

1 Answers1

1

Also, I tried to schedule the execution of a pipeline using Build, but I assume that it is a wrong approach. It rebuilds a pipeline, but I need to execute a previously published pipeline.

Assuming your python-related task runs after many other tasks, then it's not recommended to simply schedule the whole build pipeline, it will rerun the pipeline(other tasks+python script).

Only the pipeline can be scheduled the instead of tasks, so I suggest you can create a new build pipeline to run the python script. Also, a private agent is more suitable for this scenario.

Now we get two pipelines: Original A and B which used to run the python script.

  1. Set B's build completion to be A, so that if A builds successfully the first time, B will run after that.

  2. Add a command-line task or PS task as pipeline A's last task. This task(modify the yml and then push the change) will be responsible for updating the B's corresponding xx.yml file to schedule B.

  3. In this way, if A(other tasks) builds successfully, B(pipeline to run python script) will execute. And B will run daily after that successful build.

Hope it helps and if I misunderstand anything, feel free to correct me.

LoLance
  • 25,666
  • 1
  • 39
  • 73
  • I had to set `allow_reuse=False` in `PythonScriptStep`. In this case the `Schedule` works correctly. But anyway, just to comment that I want the Builds to be triggered by commits. And then the ML pipeline (not the Azure DevOps pipeline) should be executed daily, independently on commits. – Fluxy Dec 09 '19 at 08:49
  • Also, I do not understand the role of Release in all this setup. – Fluxy Dec 09 '19 at 08:49
  • Hi, sorry for the delay. For the meaning of release pipeline see [this](https://learn.microsoft.com/en-us/azure/devops/pipelines/release/?view=azure-devops). I'm bit confused about the `role of Release` you mentioned above.Hmm, I didn't find in which setup you use that release pipeline in original question :-( – LoLance Dec 11 '19 at 10:07
  • I am currently only using Build. My concern is that if it's meaningful or is it mandatory to always end up with Release? In my case, I cannot find out what exactly should I put in Release, because Build already publishes a pipeline and a schedule. – Fluxy Dec 11 '19 at 16:27
  • Sorry for the delay, I'm not sure about your real requirements, but as I know, release pipeline is designed for deployment job. And build pipeline is designed to build, test ... In these two pipelines we actually use tasks to do the real job, so if we move the deploy-related task from release pipeline to build pipeline, it actually work. But this is not a good idea, for better logic, it's recommended to use Build Pipeline to do the CI related things and CD with release pipeline. – LoLance Dec 16 '19 at 12:42