How to execute a PowerShell Command from within Azure Data Factory custom activity?

Question

I have a custom activity in Azure Data Factory, which attempts to execute the following command:

PowerShell.exe -Command "Write-Host 'Hello, world!'"

However, when I debug (run) this command from within Azure Data Factory, it runs for a long time, and finally fails.

I guess it fails because perhaps it could not locate "PowerShell.exe". How can I ensure that the ADF Custom Activity has access to PowerShell.exe?

Some sites say about specifying a package (.zip file) that contains everything needed for the exe to execute. However, since PowerShell is from Microsoft, I think it would be inappropriate to ZIP the PowerShell directory, and specify it as a package to the Custom Activity.

Please suggest as to how I can execute PowerShell command from Custom Activity of an Azure Data Factory. Thanks!

Whenever I search "Execute PowerShell from Custom Activity in Azure Data Factory", the search results are talking more about which Az PowerShell command to use to trigger start an ADF pipeline.

I saw two threads in Stackoverflow.com, where the answer just specifies to use a Custom Activity, and the answer is not specific to PowerShell command call from ADF

Here is the JSON for the task:

{ "name": "ExecutePs1CustomActivity", "properties": { "activities": [ { "name": "ExecutePSScriptCustomActivity", "type": "Custom", "dependsOn": [], "policy": { "timeout": "7.00:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "userProperties": [], "typeProperties": { "command": "PowerShell.exe -Command \"Write-Host 'Hello, world!'\"", "referenceObjects": { "linkedServices": [], "datasets": [] } }, "linkedServiceName": { "referenceName": "Ps1CustomActivityAzureBatch", "type": "LinkedServiceReference" } } ], "annotations": [] } }

I see "In Progress" for 3 minutes (180 seconds), and then it shows as "Failed."

score 0 · Answer 1 · answered Sep 10 '19 at 08:16

I would suggest you to move all you scripting task in a powershell file and copy it to a storage account linked with your custom activity. . Once done try to call it like below:

powershell .\script.ps1

You can also provide the path of the script in json like below:

{
  "name": "MyCustomActivityPipeline",
  "properties": {
    "description": "Custom activity sample",
    "activities": [{
      "type": "Custom",
      "name": "MyCustomActivity",
      "linkedServiceName": {
        "referenceName": "AzureBatchLinkedService",
        "type": "LinkedServiceReference"
      },
      "typeProperties": {
        "command": "helloworld.exe",
        "folderPath": "customactv2/helloworld",
        "resourceLinkedService": {
          "referenceName": "StorageLinkedService",
          "type": "LinkedServiceReference"
        }
      }
    }]
  }
}

Please try it and see if it helps. Also i would suggest you to troubleshoot the pipeline steps to look for detailed error.

Also to your second point "Some sites say about specifying a package (.zip file) that contains everything needed for the exe to execute." This is required when you are building a custom activity using dot net then it is must copy all the Dll's and Exe's for execution.

Hope it helps.

I will try this out tomorrow and share my observation. One thing I doubt is that the Corporate policy does not allow to keep a PS1 file in the Data Lake's file system. If I get through this hurdle, I will try your suggestion.Thanks! — Dinesh Muciliath Jayadevan, Oct 02 '19 at 13:10

How to execute a PowerShell Command from within Azure Data Factory custom activity?

1 Answers1