How to use the sops provider with terraform using an array instead an single value

Question

I'm pretty new to Terraform. I'm trying to use the sops provider plugin for encrypting secrets from a yaml file: Sops Provider

I need to create a Terraform user object for a later provisioning stage like this example:

users = [{
  name = "user123"
  password = "password12"
}]

I've prepared a secrets.values.enc.yaml file for storing my secret data:

yaml_users: 
  - name: user123
    password: password12

I've encrypted the file using "sops" command. I can decrypt the file successfully for testing purposes.

Now I try to use the encrypted file in Terraform for creating the user object:

data "sops_file" "test-secret" {
  source_file = "secrets.values.enc.yaml"
}

# user data decryption
users = yamldecode(data.sops_file.test-secret.raw).yaml_users

Unfortunately I cannot debug the data or the structure of "users" as Terraform doesn't display sensitive data. When I try to use that users variable for the later provisioning stage than it doesn't seem to be what is needed:

Cannot use a set of map of string value in for_each. An iterable collection is required.

When I do the same thing with the unencrypted yaml file everything seems to be working fine:

users = yamldecode(file("secrets.values.dec.yaml")).yaml_users

It looks like the sops provider decryption doesn't create an array or that "iterable collection" that I need.

Does anyone know how to use the terraform sops provider for decrypting an array of key-value pairs? A single value like "adminpassword" is working fine.

Answer 1

I think the "set of map of string" part of this error message is the important part: for_each requires either a map directly (in which case the map keys become the instance identifiers) or a set of individual strings (in which case those strings become the instance identifiers).

Your example YAML file shows yaml_users being defined as a YAML sequence of maps, which corresponds to a tuple of objects on conversion with yamldecode.

To use that data structure with for_each you'll need to first project it into a map whose keys will serve as the unique identifier for each instance of the resource. Assuming that the name values are suitably unique, you could project it so that those values are the keys:

data "sops_file" "test-secret" {
  source_file = "secrets.values.enc.yaml"
}

locals {
  users = tomap({
    for u in yamldecode(data.sops_file.test-secret.raw).yaml_users :
    u.name => u
  })
}

The result being a sensitive value adds an extra wrinkle here, because Terraform won't allow using a sensitive value as the identifier for an instance of a resource -- to do so would make it impossible to show the resource instance address in the UI, and impossible to describe the instance on the command line for commands that need that.

However, this does seem like exactly the use-case shown in the example of the nonsensitive function at the time I'm writing this: you have a collection that is currently wholly marked as sensitive, but you know that only parts of it are actually sensitive and so you can use nonsensitive to explain to Terraform how to separate the nonsensitive parts from the sensitive parts. Here's an updated version of the locals block in my previous example using that function:

locals {
  users = tomap({
    for u in yamldecode(data.sops_file.test-secret.raw).yaml_users :
    nonsensitive(u.name) => u
  })
}

If I'm making a correct assumption that it's only the passwords that are sensitive and that the usernames are okay to disclose, the above will produce a suitable data structure where the usernames are visible in the keys but the individual element values will still be marked as sensitive.

local.users then meets all of the expectations of resource for_each, and so you should be able to use it with whichever other resources you need to repeat systematically for each user.

Please note that Terraform's tracking of sensitive values is for UI purposes only and will not prevent this passwords from being saved in the state as a part of whichever resources make use of them. If you use Terraform to manage sensitive data then you should treat the resulting state snapshots as sensitive artifacts in their own right, being careful about where and how you store them.