1

I understand you can use a batch custom activity with custom VM image with preloaded python libraries to run your python programs through ADF.

Can we also use a plain linux VM image and load the python libraries as part of the execution of the custom activity? I understand this might increase the execution time since libraries have to be installed each time, but I want to find out how it is done. Explanation/example is appreciated.

Thanks!

Jay Gong
  • 23,163
  • 2
  • 27
  • 32
Gadam
  • 2,674
  • 8
  • 37
  • 56

1 Answers1

3

You can use Azure Batch start task to install pre defined libraries efficiently. All jobs submitted via Custom Activity against the same pool will have access to them.

Alternatively, you can install dependent libraries by executing a shell scripts against Linux pool nodes which would setup the libraries and then trigger the python code.

Abhishek Narain
  • 411
  • 2
  • 5
  • thanks, that sounds perfect. The start task seem to take in just one command, so I might have to shell script there as well if I have multiple libraries? – Gadam Nov 09 '18 at 20:37
  • that is correct. have a master script which installs all decencies. You can do this either at start task which is suggested, or even during the custom activity execution by executing the shell script, which can call the required python code (ensure all resources - Master bash script + Python file + any other resource dependency are all provided in Custom activity's folderPath property. Custom activity simply copies them on to batch node) – Abhishek Narain Nov 12 '18 at 01:45