Terraform, getting output from null_resource, local-exec and the AWS CLI

Question

I'm using Terraform to automate provision of Cognito Identity Pools in AWS. The AWS provider doesn't support Cognito yet so I've been using null_resource and local-exec to call the AWS CLI.

I have the following resource:

resource "null_resource" "create-identitypool" {
    provisioner "local-exec" {
        command = "aws cognito-identity create-identity-pool --identity-pool-name terraform_identitypool --no-allow-unauthenticated-identities --developer-provider-name login.terraform.myapp"
    }
}

which gives the following output:

null_resource.create-identitypool (local-exec): {
null_resource.create-identitypool (local-exec):     "IdentityPoolId": "eu-west-1:22549ad3-1611-......",
null_resource.create-identitypool (local-exec):     "AllowUnauthenticatedIdentities": false,
null_resource.create-identitypool (local-exec):     "DeveloperProviderName": "login.terraform.myapp",
null_resource.create-identitypool (local-exec):     "IdentityPoolName": "terraform_identitypool"
null_resource.create-identitypool (local-exec): }
null_resource.create-identitypool: Creation complete

The next step is to add some roles, which I've already created, to the identity pool:

resource "null_resource" "attach-policies-identitypool" {
    provisioner "local-exec" {
        command = "aws cognito-identity set-identity-pool-roles --identity-pool-id ${null_resource.create-identitypool.IdentityPoolId} --roles authenticated=authroleXXX,unauthenticated=unauthroleXXX"
    }
}

The issue is that I'm unable to extract the IdentityPoolId, ${null_resource.create-identitypool.IdentityPoolId}, to use in the second resource. I understand the null_resource doesn't have output attributes, so how can I get this JSON object out of the command line output. I'll also want to use tirggers and run aws cognito-identity list-identity-pools and possibly delete-identity-pool to make this all repeatable from which I'll also need the output.

Any ideas? And apologies if I've missed this information somewhere else. I've also asked this question on the Terraform mailing list, but I thought I'd try for a wider audience.

Thanks, Tim

Answer 1

There is a new data source in Terraform 0.8, external that allows you to run external commands and extract output. See data.external.

The data source should only be used for the retrieval of the Cognito data, not the execution of it. Since this is a Terraform data source, it should not have any side effects.

Answer 2

Paul's answer is correct. However, external data only works if the shell script sends data back in JSON format , which requires more work.

So, Matti Paksula made a module for this. (https://github.com/matti/terraform-shell-resource).

Using that module , we can get stdout, stderr, and exit status of ANY shell script local-exec calls.

Here is an example main.tf file. You can modify this any way you want to run any command you want including the one in your question.

#  Defining a variable , we will feed to the shell script
variable "location" { default = "us-central1-f" }


# Calling Matti's Module
module "shell_execute" {
  source  = "github.com/matti/terraform-shell-resource"
  command = "./scripts/setenv.sh"
}


# Creating a shell script on the fly
resource "local_file" "setenvvars" {
  filename = "./scripts/setenv.sh"
  content  = <<-EOT
    #!/bin/bash
    export LOCATION=${var.modinput_location}
    echo LOCATION $LOCATION
  EOT
}

#  Now, we get back the output of the script
output "shell_stdout" {
  value = module.shell_execute.stdout
}

#  Now, we get back if there are any errors
output "shell_stderr" {
  value = module.shell_execute.stderr
}

#  Now, we get back exit status of the script
output "shell_exitstatus" {
  value = module.shell_execute.exitstatus
}

Answer 3

The external data source can be used for this.

Here's a full example below.

---

Let's say that you want to run an AWS CLI command to retrieve information about a certain EC2 instance (this is just for demonstration purposes), for instance the AMI ID and the key name.

First of all, create a bash script (get_instance_details.sh) that fetches and returns the required information:

#!/bin/bash
INSTANCE_ID=$1

INSTANCE_DETAILS=$(aws ec2 describe-instances --instance-id $INSTANCE_ID)
AMI_ID=$(echo $INSTANCE_DETAILS | jq -r '.Reservations[].Instances[].ImageId')
KEY_NAME=$(echo $INSTANCE_DETAILS | jq -r '.Reservations[].Instances[].KeyName')

jq -n --arg ami_id "$AMI_ID" --arg key_name "$KEY_NAME" '{"ami_id":$ami_id,"key_name":$key_name}'

then, create a data source that runs the script:

data "external" "get_instance_details" {
  program = ["bash", "get_instance_details.sh","${aws_instance.my_instance.id}"]
}

you can then check the results with terraform console (after terraform apply):

> data.external.get_instance_details.result.ami_id
"ami-..."
> data.external.get_instance_details.result.key_name
"my-ec2-key..."

and obviously you can use those identifiers as properties for other resources.