0

Is that possible to use control M to orchestrate Azure Data factory Jobs? I found this agent that can be installed on an VM: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/bmc-software.ctm-agent-linux-arm

But I didn't find documentation about it. Cal Control M call an REST API to run and monitor a Job? I could user Azure functions and Blobs to control it.

Alex
  • 73
  • 1
  • 8

4 Answers4

0

All Control-M components can be installed and operated on Azure (and most other cloud infrastructure). Either use the link you quote or alternatively deploy Agents using Control-M Automation API (AAPI) or a combination of the two.

So long as you are on a fairly recent version Control-M you can do most operational tasks, for example you can monitor a job like so -

ctm run jobs:status::get -s "jobid=controlm:00001"

The Control-M API is developing quickly, check out the documentation linked from here -

https://docs.bmc.com/docs/automation-api/9019100monthly/services-872868740.html#Control-MAutomationAPI-Services-ctmrunjob:status::get

Also see -

https://github.com/controlm/automation-api-quickstart http://controlm.github.io https://docs.bmc.com/docs/display/public/workloadautomation/Control-M+Automation+API+-+Services https://52.32.170.215:8443/automation-api/swagger-ui.html

Mark
  • 316
  • 1
  • 5
0

At this time, I don't believe you will find any out of the box connectors for Control-M to Azure Data Factory integration. You do have some other options, though!

Proxy ADF Yourself

You can write the glue code for this, essentially being the mediator between the two.

  1. Write a program that will invoke the ADF REST API to run a pipeline. Details Here

  2. After triggering the pipeline, then write the code for monitoring for status. Details Here

  3. Have Control-M call your code via an Agent that has access to it.

    • I've done this with a C# console app running on a local server, and a Control-M Agent that invokes the glue code.
    • Control-M Documentation here also allows a way for you to execute an Azure Function directly from Control-M. This means you could put your code in an Azure Function. Details Here'

ALTERNATIVE METHOD

For a "no code" way, check out this Logic App connector. Write a logic app to run the pipeline and get the pipeline run to monitor status in a loop.

Next, Control-M should be able to use a plugin to invoke the logic app.

Notes

**Note that Control-M required an HTTP Trigger for Azure Functions and Logic Apps. **You might also be able to take advantage of the Control-M Web Services plugin. Though, in my experience, I wasn't impressed with the lack of support for different authentication methods.

Hope this helps!

Alex
  • 183
  • 1
  • 5
0

I just came across this post so a bit late to the party.

Control-M includes Application Integrator which enables you to use integrations created by others and to either enhance them or build your own. You can use REST or cli to instruct Control-M what requests should be generated to an application when a job is started, during execution and monitoring and how to analyze results and collect output.

A public repository accessible from Application Integrator shows existing jobs and there is one for Data Factory. I have extended it a bit so that the the Data Factory is started and monitored to completion via REST but then a Powershell script is invoked to retrieve the pipeline run information for each activity within the pipeline.

I've posted that job and script in https://github.com/JoeGoldberg/automation-api-community-solutions/tree/master/4-ai-job-type-examples/CTM4AzureDataFactory but the README is coming later.

Joe Goldberg
  • 38
  • 1
  • 4
0

Updated Answer: Control-M now has a fully supported integration with Azure Data Factory. Docs can be found here: https://docs.bmc.com/docs/ctm_integrations/control-m-for-azure-data-factory-994589887.html

KhanTengri
  • 1