Automated Deployment of ADF Pipelines using Azure DevOps CI/CD Pipelines

Question

I have automated the Azure ADF Pipeline Deployment process using Azure DevOps CI/CD pipelines with the help of https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment (i.e) Deploying pipelines from DEV to PROD environment ADF. I am using ARM Templates of the ADF to deploy pipelines from one environment to another. Hence I will be having a separate ARM_Parameter.json corresponding to each environment(Dev/Prod). The Problem is each ADF pipeline may have few base parameres along with it, which is not parameterized and hence it will not be available in parameter.json. Can you guys help me to replace the Dev Values with the PROD Values in Base Parameter section under each ADF Pipelines in an automated way during this automated ADF pipeline deployment process using CI/CD Pipelines?

score 1 · Answer 1 · answered Jan 14 '20 at 07:50

I see two options:

If it's only for this RUN_ENVIRONMENT parameter, you could change your parameter to variable and use the system variable @pipeline().DataFactory to determine what environment you're running in.
Otherwist, you can configure the Data Factory to generate ARM Parameters for your pipeline parameter default values, but you'll have to create a custom arm-template-parameters-definition.json file. Check the documentation here

score 1 · Answer 2 · answered Mar 11 '20 at 20:40

You could use Custom parameter with ARM template. The custom parameter for Pipeline could look like this:

"Microsoft.DataFactory/factories/pipelines": {
    "properties": {
        "parameters": {
                "RUN_ENVIRONMENT": "=:-:string"
        }
    }
},

score 0 · Answer 3 · answered Jan 14 '20 at 07:56

Replace the Dev Values with the PROD Values in Base Parameter section

Based on your screenshot, the RUN_ENVIRONMENT is the parameter of pipeline, which means while convert to ARM template, its format like:

 "resources": [
    {
      ....
      ....
      "properties": {
        "parameters": {
          "RUN_ENVIRONMENT": {
            "type": "string",
            "defaultValue": "pro"
          }
        },...      
      },...
    }
  ]

It can not be replaced by using Override template parameters in ARM deploy task. Because it will prompt The template parameters 'environment' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time.

To around this error, just install one extension and add the Replace token task into the pipeline which before ARM deploy task. And this task will replace the value of content during the build runtime:

For how to apply this task in our pipeline, you could have a refer to my answer1 and answer2

Thank You for your Comments. I thought of considering this particular agent job "Replace Tokens" under our CI/CD Pipelines, But the problem is since these are pipeline parameters & base parameters. I have the same parameter name for many pipelines, only the parameter value changes. Hence this agent job couldn't be used in that case. — IT_Guy, Jan 14 '20 at 10:19
@RJP, Ok. If this, what about configure one separate variable for one value? At this time, the specific location could be replaced with the specific value. But this will bring a big trouble while you have a lot of values. — Mengdi Liang, Jan 15 '20 at 09:50

score 0 · Answer 4 · answered May 11 '20 at 12:30

There is another approach to publish ADF, from master (collaboration) branch. You can define (replace) value for every single node (property) in JSON file (ADF object). It will resolve your problem as you can provide a separate CSV config file per each environment (stage).

Example of CSV config file (config-stage-UAT.csv):

type,name,path,value
pipeline,PL_CopyMovies,activities[0].outputs[0].parameters.BlobContainer,UAT

Then just run such cmdlet in PowerShell:

Publish-AdfV2FromJson -RootFolder "$RootFolder" -ResourceGroupName "$ResourceGroupName" -DataFactoryName "$DataFactoryName" -Location "$Location" -Stage "stage-UAT"

Check this out: azure.datafactory.tools (PowerShell module)

Automated Deployment of ADF Pipelines using Azure DevOps CI/CD Pipelines

4 Answers4