Having Terraform skip "apply" in Azure DevOps build pipeline if the resource group already exists in Azure

Question

The following YAML snippet is a part of my Azure DevOps build pipeline:

- task: TerraformTaskV2@2
  displayName: 'Terraform plan'
  inputs:
    provider: 'azurerm'
    command: 'plan'
    workingDirectory: '$(System.DefaultWorkingDirectory)/infrastructure'
    commandOptions: '-out $(Build.BuildNumber)'
    environmentServiceNameAzureRM: 'MASKED'
- task: TerraformTaskV2@2
  displayName: 'Terraform approve and apply'
  name: terraformApply
  inputs:
    provider: 'azurerm'
    command: 'apply'
    workingDirectory: '$(System.DefaultWorkingDirectory)/infrastructure'
    commandOptions: '$(Build.BuildNumber)'
    environmentServiceNameAzureRM: 'MASKED'

Also, "terraform plan" stage creates an output whose name is the same as the build number but terraform apply wouldn't pick that name to simulate a graceful skip if the resource group already exists. Terraform apply task always appends "auto-approve" per the following example where 1.0.0 is the build number:

terraform apply -auto-approve 1.0.0

This piece of YAML runs well and creates the resource group if it does not exist. There are a few other steps after this step that have to run too. terraformApply stage fails if the resource group already exists and hence the following steps won't run. I would like to have a graceful pipeline to skip terraform apply stage if the resource group already exists and execute the following steps in the pipeline after "apply". How can I achieve this goal?

The error details reads as below:

2021-06-01T13:30:26.3472705Z ##[section]Starting: Terraform approve and apply
2021-06-01T13:30:26.3481129Z ==============================================================================
2021-06-01T13:30:26.3481656Z Task         : Terraform
2021-06-01T13:30:26.3482325Z Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
2021-06-01T13:30:26.3482826Z Version      : 2.188.1
2021-06-01T13:30:26.3483165Z Author       : Microsoft Corporation
2021-06-01T13:30:26.3483587Z Help         : [Learn more about this task](https://aka.ms/AA5j5pf)
2021-06-01T13:30:26.3484057Z ==============================================================================
2021-06-01T13:30:26.4679906Z [command]/opt/hostedtoolcache/terraform/0.15.4/x64/terraform providers
2021-06-01T13:30:27.0463781Z 
2021-06-01T13:30:27.0465060Z Providers required by configuration:
2021-06-01T13:30:27.0465541Z .
2021-06-01T13:30:27.0466639Z ├── provider[registry.terraform.io/hashicorp/azurerm] >= 2.26.0
2021-06-01T13:30:27.0467592Z └── provider[registry.terraform.io/hashicorp/random]
2021-06-01T13:30:27.0467902Z 
2021-06-01T13:30:27.0478744Z [command]/opt/hostedtoolcache/terraform/0.15.4/x64/terraform validate
2021-06-01T13:30:28.5103220Z [32m[1mSuccess![0m The configuration is valid.
2021-06-01T13:30:28.5104320Z [0m
2021-06-01T13:30:28.5192981Z [command]/opt/hostedtoolcache/terraform/0.15.4/x64/terraform apply -auto-approve 1.0.0
2021-06-01T13:30:34.9083339Z [0m[1mazurerm_resource_group.gf: Creating...[0m[0m
2021-06-01T13:30:34.9697353Z [31m╷[0m[0m
2021-06-01T13:30:34.9699616Z [31m│[0m [0m[1m[31mError: [0m[0m[1mA resource with the ID "/subscriptions/MASKED/resourceGroups/FooResourceZGroup" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_resource_group" for more information.[0m
2021-06-01T13:30:34.9701090Z [31m│[0m [0m
2021-06-01T13:30:34.9702369Z [31m│[0m [0m[0m  with azurerm_resource_group.gf,
2021-06-01T13:30:34.9703361Z [31m│[0m [0m  on main.tf line 16, in resource "azurerm_resource_group" "gf":
2021-06-01T13:30:34.9704127Z [31m│[0m [0m  16: resource "azurerm_resource_group" "gf" [4m{[0m[0m
2021-06-01T13:30:34.9704722Z [31m│[0m [0m
2021-06-01T13:30:34.9705176Z [31m╵[0m[0m
2021-06-01T13:30:34.9828118Z ##[error]Error: The process '/opt/hostedtoolcache/terraform/0.15.4/x64/terraform' failed with exit code 1
2021-06-01T13:30:34.9843807Z ##[section]Finishing: Terraform approve and apply

UPDATE The terraform YAML looks like the following code snippet in the pipeline:

- task: TerraformTaskV2@2
  displayName: 'Terraform init'
  inputs:
    provider: 'azurerm'
    command: 'init'
    workingDirectory: '$(System.DefaultWorkingDirectory)/infrastructure'
    backendServiceArm: 'MASKED'
    backendAzureRmResourceGroupName: 'masked'
    backendAzureRmStorageAccountName: 'masked'
    backendAzureRmContainerName: 'multitstate'
    backendAzureRmKey: 'terraform.state'
- task: TerraformTaskV2@2
  displayName: 'Terraform plan'
  inputs:
    provider: 'azurerm'
    command: 'plan'
    workingDirectory: '$(System.DefaultWorkingDirectory)/infrastructure'
    commandOptions: '-out $(Build.BuildNumber)'
    environmentServiceNameAzureRM: 'MASKED'
- task: TerraformTaskV2@2
  displayName: 'Terraform approve and apply'
  name: terraformApply
  inputs:
    provider: 'azurerm'
    command: 'apply'
    workingDirectory: '$(System.DefaultWorkingDirectory)/infrastructure'
    commandOptions: '$(Build.BuildNumber)'
    environmentServiceNameAzureRM: 'MASKED'

Where is your terraform state file stored? If the resource already exists, then it should already be in the terraform state file, so terraform shouldn't be trying to create it again. — Mark B, Jun 03 '21 at 14:02
So why is terraform failing if the resource already exists? Is the resource in the state or not? You haven't included any details about the actual failure, for example the error message, in your question. — Mark B, Jun 03 '21 at 14:07
@MarkB I updated the question by quoting the failure message. — Arash, Jun 03 '21 at 14:35
""/subscriptions/MASKED/resourceGroups/FooResourceZGroup" already exists - to be managed via Terraform this resource needs to be imported into the State." Why is the resource not managed by terraform? — Mark B, Jun 03 '21 at 16:31

Krzysztof Madej · Answer 1 · 2021-06-04T11:36:35.460

0

Since your resource was created out of Terraform you need to import into terraform. Basically this is not done without your clear instruction. And for that you need to use terraform import command. You need to run it only once. So you can create a seprate pipeline, run it one and the remove this pipeline (or do this as a step in current pipeline).

What is bad that this task doesn't support import command:

So you need to run custom script like:

terraform init
terraform state pull
terraform import azurerm_resource_group.example /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example

Please remove resource, configure backend for your tasks:

    backendServiceArm: 'full-subscription'
    backendAzureRmResourceGroupName: 'azure-functions-minimal-downtime'
    backendAzureRmStorageAccountName: 'afmdst'
    backendAzureRmContainerName: 'activityitems'
    backendAzureRmKey: 'test'

and run your pipeline again.

edited Jun 04 '21 at 11:36

answered Jun 03 '21 at 16:38

Krzysztof Madej

32,704
10
78
107

I guess I am missing a point. The terraform is creating the resource group and the resources. So, why have you assumed that the resources were created outside of terraform? – Arash Jun 04 '21 at 01:40
This message `""/subscriptions/MASKED/resourceGroups/FooResourceZGroup" already exists - to be managed via Terraform this resource needs to be imported into the State."` says this. Basically, if terrafrom creates resource it keeps information about this in state. If thre is no info about resource in state for terraform it means that it was created outside of terraform. – Krzysztof Madej Jun 04 '21 at 01:50
Understood. Terraform's state is supposed to be retained in an azure storage account backend, so the problem is that why it does not retain it in there! – Arash Jun 04 '21 at 11:25
It could be this. Difficult to say, I don't see you backend configuration. Actually you don't have `backend***` settings defined in your tasks. – Krzysztof Madej Jun 04 '21 at 11:34
Just added the YAML to the question. Please see the "UPDATE" section. thanks – Arash Jun 04 '21 at 11:41
Please delete your resource (if you could) and run again your pipeline. WIth configured backend you should be fine. If you can't delete you need to import resource into state as I already shown above. – Krzysztof Madej Jun 04 '21 at 12:38
I already tried it several times but it was not conclusive. Apparently, the only way would be to import! – Arash Jun 04 '21 at 12:42

Having Terraform skip "apply" in Azure DevOps build pipeline if the resource group already exists in Azure

1 Answers1