1

Azure Data Factory can execute custom activities as Batch Service Jobs. These jobs can run from an .exe (and associated dependencies) in a storage account which are copied across prior to execution.

There is a limitation on the files in the storage account that can be used:

Total size of resourceFiles cannot be more than 32768 characters

The solution appears to be to zip the files in the storage account and unzip as part of the command. This post suggests running the Batch Service Command in Azure Data Factory as:

Unzip.exe [myZipFilename] && MyExeName.exe [cmdLineArgs]

Running this locally on a Windows 10 machine works fine. Setting this as the Command parameter on the batch service custom activity (using a Cloud Services Windows Server 2019 OS Image App Pool) results in:

caution: filename not matched: &&

It feels like something basic that I'm missing but I've tried various permutations and cannot get it to work.

Cargowire
  • 1,488
  • 10
  • 21
  • Instead of zipping and unzipping files, you may also want to consider creating the resource files with the `storageContainerUrl` property, rather than specifying each individuality. Assuming some of the files are in the same container, this should drastically reduce the number of characters in your request. – brklein May 06 '19 at 14:56
  • @brklein I'm not sure I understand what you mean? In the Data Factory Custom activity I connect to the batch service and app pool then in the settings set a 'Resource linked service' to the storage account and container with a folder path for it to copy the application from. This is as granular as it appears to be? whatever it is doing behind the scenes when the activity runs to retrieve those files and put them on the batch node is essentially a black box to me. – Cargowire May 06 '19 at 21:54
  • That is my bad. I didn't realize you were using ADF which only exposes a subset of Batch functionality. – brklein May 07 '19 at 22:14

4 Answers4

3

Sharing few possibilities which could be happenening, I agree that something small is missing, if you want to share the exact commandline you are trying then feel free to share.

Few ideas which could be causing this behavioour:

  • Please try your command-line under quotes from batch point of view like: cmd /c "Unzip.exe [myZipFilename] && MyExeName.exe [cmdLineArgs]"

  • Make sure your file exist i.e. one possibility is : that in the end the command is trying to run an empty string with && i.e. cmd /c "unzip.exe "empty" && ...

Hope one of the above 2 fixes, or feel free to add more detail et. al.

Tats_innit
  • 33,991
  • 10
  • 71
  • 77
  • I'm unsure of the context in which commands are run on the Azure Batch Node Agent but Unzip.exe direct as the command executes (and you can see the default help output) but running it as part of `cmd /c "Unzip.exe"` fails with unzip.exe being unknown. – Cargowire May 06 '19 at 22:41
  • 1
    Exactly, I did wonder so why do you think that `unzip.exe` will work is it something ADF bake it in? We can discuss and see what will actually be helpful, but I am not sure about how ADF works **Idea**: when you run the node, remove log inside the node and then you can run this command in the cmd prompt there, if that works then this should, otherwise you need to first make sure the resource you are using is something which is baked inside the node before use. i.e. I am unsure if `unzip.exe` comes baked inside our nodes or not. – Tats_innit May 07 '19 at 01:16
  • 1
    I ended up giving up on the idea that unzip was somehow baked into the execution context and included it as part of my resources. – Cargowire May 08 '19 at 10:51
2

Without full knowledge of the context in which ADF runs Custom Activity Commands on a Windows Batch Service Node I changed my setup to avoid expecting Unzip.exe to exist (which it appears not to when running under cmd /c "Unzip.exe" rather than with just Unzip.exe as the command).

Now my storage account contents backing the custom activity has:

  • executable.zip (my .NET Core Console application published for windows with all dependencies)
  • unzip.exe (taken from Git Bash on my local machine)
    • including the msys-2.0.dll and msys-bz2-1.dll dependencies

The command in ADF is then:

cmd /c "Unzip.exe executable-with-deps.zip && executable.exe"
Cargowire
  • 1,488
  • 10
  • 21
0

I had the same problem and solved it with a short powershell script, that extracts my zip and start my Program. The generic script look like (called unzipandrun.ps1):

[CmdletBinding()]
param (    
    [Parameter(Mandatory = $true)][String]$zipFile,    
    [Parameter(Mandatory = $true)][String]$executable,    
    [Parameter(Mandatory = $true)][String]$prm
)
#######
# Unzip
#######
Add-Type -AssemblyName System.IO.Compression.FileSystem
function Unzip
{
    param([string]$zipfile, [string]$outpath)
    [System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath)
}
"Start Unzip $zipFile" | Write-Host
Unzip $zipFile $pwd
###########
# Start exe
###########
"Start $executable $prm" | Write-Host
$cmd = "& $executable $prm"
Invoke-Expression $cmd

So if I want to start myapp.exe with parameter /t 123 on azure batch I would start:

PowerShell -NoProfile -ExecutionPolicy Bypass -Command "unzipandrun.ps1 '/t 123'"
Daniel W.
  • 938
  • 8
  • 21
0

Another option, which doesn't require copying additional files along with zip file:

cmd /c "tar -xf MyExe.zip & MyExe.exe"

tar -xf MyExe.zip & MyExe.exe gives an error, but cmd /c "tar -xf MyExe.zip & MyExe.exe" works fine on windows 10

Olga
  • 49
  • 5