Terrafrom Init is failing to locate module when providing relative path

Question

I've started building an infrastructure using terraform. Within that TF configuration I am calling a module to be used by using relative path. This is successful in a classic release but I have been tasked with converting the pipeline to yaml. When I run terraform init step the agent finds the Tf config files but can't find the modules folder even though the artifact was downloaded on a previous task.

yaml file:

trigger:
- master

resources:
  pipelines:
  - pipeline: Dashboard-infra
    project: Infrastructure
    source: IT Dashboard
  - pipeline: Infra-modules
    project: Infrastructure
    source: AWS Modules
    trigger: true

stages:
- stage: Test
  displayName: Test
  variables:
  - group: "Non-Prod Keys"
  jobs:
    - deployment:
      displayName: string
      variables:
       region: us-east-1
       app_name: it-dashboard
       environment: test
       tf.path: 'IT Dashboard'
      pool:
        vmImage: 'ubuntu-latest'
      environment: test
      strategy:
        runOnce:
         deploy:
            steps:
            - task: DownloadBuildArtifacts@1
              inputs:
                buildType: 'specific'
                project: '23e9505e-a627-4681-9598-2bd8b6c1204c'
                pipeline: '547'
                buildVersionToDownload: 'latest'
                downloadType: 'single'
                artifactName: 'drop'
                downloadPath: '$(Agent.BuildDirectory)/s'

            - task: DownloadBuildArtifacts@1
              inputs:
                buildType: 'specific'
                project: '23e9505e-a627-4681-9598-2bd8b6c1204c'
                pipeline: '88'
                buildVersionToDownload: 'latest'
                downloadType: 'single'
                artifactName: 'Modules'
                downloadPath: '$(agent.builddirectory)/s'
            - task: ExtractFiles@1
              inputs:
                archiveFilePatterns: 'drop/infrastructure.zip'
                destinationFolder: '$(System.DefaultWorkingDirectory)'
                cleanDestinationFolder: false
                overwriteExistingFiles: false
                
            - task: ExtractFiles@1
              inputs:
                archiveFilePatterns: 'Modules/drop.zip'
                destinationFolder: '$(System.DefaultWorkingDirectory)'
                cleanDestinationFolder: false
                overwriteExistingFiles: false
            - task: TerraformInstaller@0
              inputs:
                terraformVersion: '0.12.3'
            - task: TerraformTaskV2@2
              inputs:
                provider: 'aws'
                command: 'init'
                workingDirectory: '$(System.DefaultWorkingDirectory)/$(tf.path)'
                commandOptions: '-var "region=$(region)" -var "app_name=$(app.name)" -var "environment=$(environment)"'
                backendServiceAWS: 'tf_nonprod'
                backendAWSBucketName: 'wdrx-deployments'
                backendAWSKey: '$(environment)/$(app.name)/infrastructure/$(region).tfstate'

Raw error log:

2021-10-29T12:30:16.5973748Z ##[section]Starting: TerraformTaskV2
2021-10-29T12:30:16.5981535Z ==============================================================================
2021-10-29T12:30:16.5981842Z Task         : Terraform
2021-10-29T12:30:16.5982217Z Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
2021-10-29T12:30:16.5982555Z Version      : 2.188.1
2021-10-29T12:30:16.5982791Z Author       : Microsoft Corporation
2021-10-29T12:30:16.5983122Z Help         : [Learn more about this task](https://aka.ms/AA5j5pf)
2021-10-29T12:30:16.5983461Z ==============================================================================
2021-10-29T12:30:16.7253372Z [command]/opt/hostedtoolcache/terraform/0.12.3/x64/terraform init -var region=*** -var app_name=$(app.name) -var environment=test -backend-config=bucket=wdrx-deployments -backend-config=key=test/$(app.name)/infrastructure/***.tfstate -backend-config=region=*** -backend-config=access_key=*** -backend-config=secret_key=***
2021-10-29T12:30:16.7532941Z [0m[1mInitializing modules...[0m
2021-10-29T12:30:16.7558115Z - S3-env in ../Modules/S3
2021-10-29T12:30:16.7578267Z - S3-env.Global-Vars in ../Modules/Global-Vars
2021-10-29T12:30:16.7585434Z - global-vars in 
2021-10-29T12:30:16.7589958Z [31m
2021-10-29T12:30:16.7597321Z [1m[31mError: [0m[0m[1mUnreadable module directory[0m
2021-10-29T12:30:16.7597847Z 
2021-10-29T12:30:16.7599087Z [0mUnable to evaluate directory symlink: lstat ../Modules/global-vars: no such
2021-10-29T12:30:16.7599550Z file or directory
2021-10-29T12:30:16.7599933Z [0m[0m
2021-10-29T12:30:16.7600324Z [31m
2021-10-29T12:30:16.7600779Z [1m[31mError: [0m[0m[1mFailed to read module directory[0m
2021-10-29T12:30:16.7600986Z 
2021-10-29T12:30:16.7601405Z [0mModule directory  does not exist or cannot be read.
2021-10-29T12:30:16.7601808Z [0m[0m
2021-10-29T12:30:16.7602135Z [31m
2021-10-29T12:30:16.7602573Z [1m[31mError: [0m[0m[1mUnreadable module directory[0m
2021-10-29T12:30:16.7602768Z 
2021-10-29T12:30:16.7603271Z [0mUnable to evaluate directory symlink: lstat ../Modules/global-vars: no such
2021-10-29T12:30:16.7603636Z file or directory
2021-10-29T12:30:16.7603964Z [0m[0m
2021-10-29T12:30:16.7604291Z [31m
2021-10-29T12:30:16.7604749Z [1m[31mError: [0m[0m[1mFailed to read module directory[0m
2021-10-29T12:30:16.7604936Z 
2021-10-29T12:30:16.7605370Z [0mModule directory  does not exist or cannot be read.
2021-10-29T12:30:16.7605770Z [0m[0m
2021-10-29T12:30:16.7743995Z ##[error]Error: The process '/opt/hostedtoolcache/terraform/0.12.3/x64/terraform' failed with exit code 1
2021-10-29T12:30:16.7756780Z ##[section]Finishing: TerraformTaskV2

I have attempted to even move the modules folder inside the tf.path so it is within the same folder as the tf config files and changed the location from "../" to "./". No matter what repo I extract the modules folder to (after downloading as artifact from another build pipeline) it cannot be found when calling it on the tf config files. I am fairly new to DevOps and would appreciate any help or just being pointed in the right direction.

a little hint: if you add `-no-color` to terraform commandline, you'll get rid of all those `[31m` and the log will be clearer. — qbik, Oct 30 '21 at 04:44

score 0 · Answer 1 · answered Oct 30 '21 at 04:59

Define system.debug: true variable at global level to enable debug logs - maybe something there will give you a hint:
```
variables:
  system.debug: true
```
Apart from downloaded artifacts, do you expect to have files checked out from the repo the pipeline is defined in? The deployment job doesn't checkout git files by default, so you may want to add checkout: self to steps there.
Unable to evaluate directory symlink: lstat ../Modules/global-vars - this is suspicious, I wouldn't expect any symlinks in there. But maybe the error message is just misleading.
A useful trick is to log the whole directory structure.

You can do this with a bash script step (might need to apt install tree first):
```
 - script: tree
```
Or with powershell (will work on MS-hosted linux agent):
```
- pwsh: Get-ChildItem -Path '$(agent.builddirectory)' -recurse
```

Terrafrom Init is failing to locate module when providing relative path

1 Answers1