Error 403, "The caller does not have permission" when trying to impersonate Service Account in GCP

Question

I have this fairly easy Terraform configuration:

terraform {
  required_providers {
    google = { 
      source  = "hashicorp/google"
      version = "~> 4.19.0"
    }   

    airflow = { 
      source = "DrFaust92/airflow"
      version = "~> 0.6.0"
    }   
  }
}

data "http" "client_id" {
  url = var.cloud_composer.airflow_uri
}

resource "google_service_account" "impersonated-gsa" {
  account_id = "impersonated"
}

data "google_service_account_access_token" "impersonated" {
  target_service_account = google_service_account.impersonated-gsa.email
  delegates              = []
  scopes                 = ["userinfo-email", "cloud-platform"]
  lifetime               = "300s"
}

provider "google" {
  alias        = "impersonated"
  access_token = data.google_service_account_access_token.impersonated.access_token
}

data "google_service_account_id_token" "oidc" {
  provider               = google.impersonated
  target_service_account = google_service_account.impersonated-gsa.email
  delegates              = []
  include_email          = true
  target_audience        = regex("[A-Za-z0-9-]*\\.apps\\.googleusercontent\\.com", data.http.client_id.body)
}

provider "airflow" {
  base_endpoint = data.http.client_id.url
  oauth2_token = data.google_service_account_id_token.oidc.id_token
}

The configuration is mostly directly copied from the documentation of Terraform Airflow provider.

When I try to run Terraform on it, say for example terraform plan, I get this 403 error:

╷
│ Error: googleapi: Error 403: The caller does not have permission, forbidden
│ 
│   with module.airflow.data.google_service_account_access_token.impersonated,
│   on modules/airflow/providers.tf line 23, in data "google_service_account_access_token" "impersonated":
│   23: data "google_service_account_access_token" "impersonated" {
│ 
╵

I am running Terraform in my local terminal, being authenticated with my gcloud user as stated in Terraform docs. My user has actual Owner role in GCP, which should require no more permissions to do this job, as far as I know. Anyway, I added myself the roles/iam.serviceAccountTokenCreator role as well, which seems needed to perform this operation, but didn't help and the error is the same.

At this point, I have no idea what could I be doing wrong, or even if the Terraform configuration itself is correct.

Can someone give any advice or suggestion? Thanks.

Edit to add more details because I can't put an entire answer to John Hanley comment:

Right,

1. I haven't see userinfo-email in any doc. Actually here, here and here is always userinfo-email. Anyway, I changed to userinfo.email. This didn't affect the results.
2. I checked that to exhaustion, and indeed my user account, the only one I had configured to run Terraform, is an Owner. There is no chance I am running Terraform with any other non-Owner user.
3. Okay, so to triple check now I manually created a Service Account in GCP dashboard, assigned it Owner role, also assigned it Service Account Token Creator role. Then I configured my local to run Terraform with that, and the error is exactly the same.

So, the mistake must be in the TF configuration.

Answer 1

1. The scope userinfo-email is invalid. It should be userinfo.email.

2. You state that you are using an account with the Owner role. However, the error message indicates that you are not using that identity with Terraform.

3. Using a user identity can result in errors (API quotas). You should be using a service account. I recommend allocating a service account, assigning it the required roles, and initializing the Terraform provider with the service account.

Answer 2

Your terminal might be configured with Environment Variables, which I had the same issue with. GOOGLE_IMPERSONATE_SERVICE_ACCOUNT env was configured in my case, which caused the issue.