2

I want to use terraform to manage my Cloud Run deployments but the Cloud Run deployments depend on secrets. Therefore Secret Manager secrets have to be created by terraform beforehand but I do not want my secrets stored in variables.

What I have

  • am using gcp cloud storage to store tfsate
  • deploys cloud run that needs secrets
  • secrets have to exist or cloud run fails to start
  • can only use google_secret_manager_secret_version if the value is defined in tfvars
  • don't want secrets values in plain text in any local files
  • creating google_secret_manager_secret_version with an initial random value, will replace on terrafrom apply after manually changing secret to real value
  • can't only create google_secret_manager_secret without having a google_secret_manager_secret_version because cloud run needs a latest value to be present to be created.

All the tutroials I found use google_secret_manager_secret_version with a hardcoded secret in variables. That is a NO, having secrets locally in files is a security problem.

The solution I would like to do is to have terraform create the secrets with random values and not touch them afterwards. Then I can replace them with real values through the GCP ui. This is not working because terraform wants to keep the secrets values in sync with the initialed random values. lifecycle ignore_changes does not work because the secrets are never updated but deleted and created.

Has anybody come up with a good approach of using terraform to create a service that depends on secrets but not have to have secrets be defined in any local files?

mlablablab
  • 220
  • 2
  • 9
  • 1
    Separate the creation of the secrets from the services that use the secrets. – John Hanley May 01 '23 at 19:49
  • 1
    Most of the time, the secrets are handle/set by humans (devops). You could have solution like Hashicorp vault, but it's also not magic. If you have an API key to set, and you can get this key from a 3rd party website, human in the loop is at least required to initiate the secret. – guillaume blaquiere May 01 '23 at 20:35
  • @JohnHanley are you proposing to ensure the secrets are already created before running 'terraform init'? It makes sense, my thought was since Terraform handles all the service enabling and knows what secrets it needs to attach to cloud run, it would have been great to have it seed the secrets on start. Am going to update a bash script I already use to create the state bucket to also check for presence of secrets. – mlablablab May 01 '23 at 21:21
  • @guillaumeblaquiere thanks for clarifying, what is the purpose of `google_secret_manager_secret` and `google_secret_manager_secret_version` then? Is it to store values into secretes manager that terraform generates? – mlablablab May 01 '23 at 21:27
  • 2
    Yes, separate the creation/management of secrets from the service. The Terraform HCL does not need to know the value of the secrets only their name/version. The security issue is storing the values of the secrets in the Terraform state files. The management issue is trying to change secrets without breaking Terraform's state. – John Hanley May 01 '23 at 23:15

1 Answers1

2

Thank you for the help

got it working with a bash script that sets up secrets in GCP, and imports only google_secret_manager_secret into terraform so terraform only know that there are secrets but will not keep track of their values.

After which I can update the secrets with versions of actual values the system will use without terraform deleting secret values on changes.

here is a repo with what worked

https://github.com/firemuzzy/gcp-terraform-bootstrap-secrets

Two key parts are pulled out below: the script that seeds and imports the secrets, the tf file defining the secrets resource

gcloud.init.sh

#
# abridged code, of gcloud.init.sh
# incomplete, see repo for full context
#
# seeds the env values and imports them into terraform
#
#!/bin/sh

# Secrets that need to be seeded and then importeted
SECRETS=( "SECRET_1" "SECRET_2" "SECRET_3" )

# .... parse command line args, set up state ...

#
# seed gcp secrete manager value with a random string if it is not there
#
on_missing_secret_value_fill_w_random() { .... }


#
# SEED SECRETS
#
echo ""
echo "Seeding secrets:"

SEEDED_SECRET=0
for SECRET in ${SECRETS[*]}
do
  on_missing_secret_value_fill_w_random ${GCP_PROJECT_ID} ${SECRET}
  if [[ $? -eq 0 ]]; then
    SEEDED_SECRET=1
  fi
done

if [[ $SEEDED_SECRET -eq 1 ]]; then
  echo ""
  echo "⚠️  UPDATE secrets with real values through Secrets Manger UI"
  echo "   missing secrets were seeded with random values"
  echo "   https://console.cloud.google.com/security/secret-manager?project=${GCP_PROJECT_ID}"
fi

echo ""
terraform init
echo ""

#
# IMPORT SEEDED SECRETS, 
# will error that it already has values if ran mulitple time
# its does not really break anything, not handling it
#
echo ""
echo "Importing secrets:"
echo ""

for SECRET in ${SECRETS[*]}
do
  terraform import "google_secret_manager_secret.my_service[\"${SECRET}\"]" ${SECRET}
done

secrets.tf

#
# secrets.tf - defines the secrets resource
#
resource "google_secret_manager_secret" "my_service" {

  # loop over all the secrets we need to have
  for_each  = toset(local.secret_keys)

  secret_id = each.key

  replication {
    automatic = true
  }

  depends_on = [google_project_service.secretmanager]
}
mlablablab
  • 220
  • 2
  • 9