Continuous integration and delivery for an Azure Synapse Analytics workspace using DevOps

Question

I am using Azure DevOps for continuous integration and delivery for an Azure Synapse Analytics workspace where I have deploy the workspace to Development, Test, PreProduction and Production environments. The pipelines use parametrized LinkedServices, Datasets and Trigger. I am following these Microsoft documents document1, document2.

I have these different artifacts that where I need to pass parameters from the yaml file using the TemplateParametersForWorkspace.json.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "workspaceName": {
            "value": "devSynapse"
        },
        "LinkedService_connectionString": {
            "value": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=;Initial Catalog="
        },
        "Trigger_Test_parameters_server": {
            "value": "devServer"
        }
    }
}

I need you to point me in right direction on how to pass variables from DevOps CI/CD to change the values according to the environment the Synpase workspace is deployed to?

score 1 · Answer 1 · answered Feb 18 '23 at 07:15

Approach 1)

You can add Environment variable in the ARM template and use it for different environments in your pipeline like below:-

"connectionstring": {

"type": "string"

},

"environment": {

"type": "string"

}

},

"variables": {

"storageBlobDataContributorRoleID": "ba92f5b4-2d11-453d-a403-e96b0029c9fe",

"defaultDataLakeStorageAccountUrl": "[concat('https://', parameters('defaultDataLakeStorageAccountName'), '.dfs.core.windows.net')]",

"connectionString": "[if(equals(parameters('environment'), 'Production'), parameters('connectionStringProd'), parameters('connectionStringNonProd'))]"

Output:-

enter image description here

The above parameter checks, If the environment variable is set to Production, then it uses Production connection string or else other environment.

Approach 2)

You can also mention each production in the Azure DevOps Pipeline variables yaml script and use it in the Azure Pipelines like below:-

trigger:
- main

variables:
  - name: DevEnvironment
    value: "env-url"
  - name: TestEnvironment
    value: "env-url"
  - name: PreProdEnvironment
    value: "env-url"
  - name: ProdEnvironment
    value: "env-url"

stages:
- stage: Build
  displayName: 'Build stage'
  jobs:
  - job: Build
    displayName: 'Build job'
    steps:
    - script: echo 'Building the Synapse Analytics workspace...'
      displayName: 'Build step'

- stage: DeployToDev
  displayName: 'Deploy to Development'
  dependsOn: Build
  condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
  jobs:
  - job: Deploy
    displayName: 'Deploy job'
    steps:
    - task: AzureResourceGroupDeployment@2
      inputs:
        deploymentScope: 'Resource Group'
        azureResourceManagerConnection: 'MyAzureResourceManagerConnection'
        subscriptionId: '<subscription-id>'
        action: 'Create Or Update Resource Group'
        resourceGroupName: '<resource-group-name>'
        location: '<resource-group-location>'
        templateLocation: 'Linked artifact'
        csmFile: '<path-to-arm-template>'
        csmParametersFile: '<path-to-arm-template-parameters>'
        overrideParameters: '-environment $(DevEnvironment)'

- stage: DeployToTest
  displayName: 'Deploy to Test'
  dependsOn: Build
  condition: and(eq(variables['Build.SourceBranch'], 'refs/heads/main'), eq(variables['Build.Reason'], 'Schedule'))
  jobs:
  - job: Deploy
    displayName: 'Deploy job'
    steps:
    - task: AzureResourceGroupDeployment@2
      inputs:
        deploymentScope: 'Resource Group'
        azureResourceManagerConnection: 'MyAzureResourceManagerConnection'
        subscriptionId: '<subscription-id>'
        action: 'Create Or Update Resource Group'
        resourceGroupName: '<resource-group-name>'
        location: '<resource-group-location>'
        templateLocation: 'Linked artifact'
        csmFile: '<path-to-arm-template>'
        csmParametersFile: '<path-to-arm-template-parameters>'
        overrideParameters: '-environment $(TestEnvironment)'

- stage: DeployToPreProd
  displayName: 'Deploy to Pre-Production'
  dependsOn: Build
  condition: and(eq(variables['Build.SourceBranch'], 'refs/heads/main'), eq(variables['Build.Reason'], 'Manual'))
  jobs:
  - job: Deploy
    displayName: 'Deploy job'
    steps:
    - task: AzureResourceGroupDeployment@2
      inputs:
        deploymentScope: 'Resource Group'
        azureResourceManagerConnection: 'MyAzureResourceManagerConnection'
        subscriptionId: '<subscription-id>'
        action: 'Create Or Update Resource Group'
        resourceGroupName: '<resource-group-name>'
        location: '<resource-group-location>'
        templateLocation: 'Linked artifact'
        csmFile: '<path-to-arm-template>'
        csmParametersFile: '<path-to-arm-template-parameters>'
        overrideParameters: '-environment $(PreProdEnvironment)'

- stage: DeployToProd
  displayName: 'Deploy to Production'
  dependsOn: Build
  condition: and(eq(variables['Build.SourceBranch'], 'refs/heads/main'), eq(variables['Build.Reason'], 'Manual'))
  jobs:
  - job: Deploy
    displayName: 'Deploy job'

Approach 3)

Alternatively you can also configure different stages Dev, Test, Prod, Pre-Prod to deploy your Synapse workspace in specific environment like below:-

enter image description here

@SiddeshDesai, Thanks for your response. I'm still unclear how do I pass different variable values for different Synapse artifacts based on which Synapse environment (Development, Test, PreProduction and Production) the artifacts are deployed to using TemplateParametersForWorkspace.json in the yaml file. — paone, Feb 27 '23 at 01:51
@SiddeshDesai, Is there a way I could declare variables in the yaml file and use these variables else where across different synapse workspace artifacts for instance datasets, linked services or triggers checked into Azure devops git repo during deployment? — paone, Feb 27 '23 at 05:17
You can save your variables in a Variable group and use it via yaml script in any pipeline repo within your Azure DevOps project — SiddheshDesai, Mar 01 '23 at 04:28
Hi @SiddeshDesai, I created a variable group dev for my dev release pipeline and added a new variable `myserver`. Now I reference this `echo $(myserver)` in my `build.yaml` file I see the values propogate but when I replace `"devServer"` with `"$(myserver)"` in the TemplateParametersForWorkspace.json file after the deployment I don't see the actual dev server name instead see `$(myserver)`. Could you please let me know how to reference these variable values in other json files in my repo. — paone, Mar 01 '23 at 21:28

Continuous integration and delivery for an Azure Synapse Analytics workspace using DevOps

1 Answers1