2

I want to automatically deploy artifacts (custom python packages / .whl files) from the Azure DevOps artifact feed to a Synapse spark pool.

Currently I have to manually:

  • download the .whl file from the artifact feed
  • upload the .whl file to the synapse workspace
  • add the package/.whl to the packages of a spark pool (> "Select from workspace packages").

I have so far not found any option to do this as part of a release/pipeline in Azure DevOps or via the AzureCLI and also have not found any documentation on it. I was wondering whether anyone has found a solution concerning automating this step? It is quite cumbersome to have to do it manually.

PS: I already asked this question on the MS forum, but have gotten no official answer yet.

Cribber
  • 2,513
  • 2
  • 21
  • 60
  • 1. Download package from DevOps feed with task: [Download package](https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/download-package?view=azure-devops). 2. Use "Synapse workspace deployment" task to deploy the package. Please check link: https://learn.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-delivery#set-up-a-stage-task-for-azure-synapse-artifacts-deployment – wade zhou - MSFT Jun 23 '22 at 06:47
  • The second link describes how to deploy artifacts such as pipelines, scripts, linkedServices or triggers within a Synapse Workspace via configuration of the JSON template - but nowhere does it say anything about deploying packages to spark pools... how is this supposed to help? – Cribber Jun 23 '22 at 08:40

1 Answers1

5

I managed to update the pool spark packages by using both the Download Packages task and the Azure CLI task in the release pipeline on devops.

First use the Download Packages task to download your feed to the pipeline $(System.DefaultWorkingDirectory) then use the Azure CLI to update the sparkpool.

In the Azure CLI task write in the inline command field the two following lines. Use the $(System.DefaultWorkingDirectory) as working directory, also I recommand to check the Use global Azure CLI configuration checkbox.

  • First command to upload your .whl to the workspace. Here is the doc

    az synapse workspace-package upload --workspace-name yourworkspace --package yourpackage.whl

  • Then upload it to the pool spark. Here is the doc.

    az synapse spark pool update --name yourpoolspark --workspace-name yourworkspace --resource-group yourresourcegroupe --package-action Add --package yourpackage.whl

Those two commands will add the package to your workspace and then upload it to the poolspark.

You can achieve the same things with the Azure Powershell task on a Windows agent. Here is the doc to explore : link.

Hope it will help you !

SneakyMoon
  • 66
  • 3
  • Do you have another tip on how to pass the name of the latest package to the Azure CLI task? I can set the version to `latest` in the DownloadPackage task, but I need to reference the specific file name in the Azure CLI – Cribber Jul 11 '22 at 11:25
  • right now I managed to do it by using `$file_name = Get-ChildItem *.whl` and `echo $file_name.name` , but Im sure there is a better way – Cribber Jul 11 '22 at 11:58
  • 2
    What you Can do is convert the release pipeline in a build pipeline by copying the yaml code of the tasks and add a new stage. By doing that you can access the package name as a variable. If the pipeline producing your feed is a build pipeline (with yaml code) you can merge the yaml codes and share the feed name between the yaml stages – SneakyMoon Jul 12 '22 at 20:11
  • 1
    Ohh, fancy idea. Thanks! – Cribber Jul 13 '22 at 07:12