0

I use Artifactory to publish NuGet packages. Package would have a file in publish folder which I should copy to Databricks cluster using Octopus deploy but can't seems to find a way to create step template

If I use deploy package step I can copy files/folder to target and read file like below

Get-ChildItem -Path "#{DATABRICKS_WORKING_DIRECTORY}" -Recurse -Filter *.whl | ForEach-Object  {
    $_.fullname
    #code to push file to databricks
}

Above approach might work but doesn't seems to be best approach, it would have been nice if I can create step template to push the file, any suggestion is highly appreciated.

TylerH
  • 20,799
  • 66
  • 75
  • 101
Sudama Tripathi
  • 349
  • 1
  • 5
  • 20

1 Answers1

0

I'm not familiar with DataBricks, however, I did a little research and I think you could create a template using the Run a Script task as the base template. Add a Package Parameter for the file that contains the .whl file, then use https://docs.databricks.com/dev-tools/api/latest/libraries.html#install to upload it to DataBricks. Does that help?

Twerthi
  • 36
  • 4
  • Run a script is something I tried but I have dependenacy on the package which should be downloaded and extracted on the target server before I could be copied, are you suggesting download and extraction also inside the script? – Sudama Tripathi Mar 08 '23 at 20:55
  • Not necessarily. The Run a Script Task can be run on a worker (https://octopus.com/docs/infrastructure/workers) instead of a target. In this case, the package would be extracted on the worker, then can be uploaded to DataBricks. – Twerthi Mar 08 '23 at 21:11