2

I am trying to spin up GCP Cloud Composer using the below set of terraform script code base:

resource "google_composer_environment" "test" {
  name   = "example-composer-env-tf-c2"
  region = "us-central1"
  config {

    software_config {
      image_version = "composer-2-airflow-2"
    }

    workloads_config {
      scheduler {
        cpu        = 0.5
        memory_gb  = 1.875
        storage_gb = 1
        count      = 1
      }
      web_server {
        cpu        = 0.5
        memory_gb  = 1.875
        storage_gb = 1
      }
      worker {
        cpu        = 0.5
        memory_gb  = 1.875
        storage_gb = 1
        min_count  = 1
        max_count  = 3
      }


    }
    environment_size = "ENVIRONMENT_SIZE_SMALL"

    node_config {
      network         = google_compute_network.test.id
      subnetwork      = google_compute_subnetwork.test.id
      service_account = google_service_account.test.name
    }
  }
}

resource "google_compute_network" "test" {
  name                    = "composer-test-network3"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "test" {
  name          = "composer-test-subnetwork"
  ip_cidr_range = "10.2.0.0/16"
  region        = "us-central1"
  network       = google_compute_network.test.id
}

resource "google_service_account" "test" {
  account_id   = "composer-env-account"
  display_name = "Test Service Account for Composer Environment"
}

resource "google_project_iam_member" "composer-worker" {
  project = "inlaid-ally-373906"
  role    = "roles/composer.worker"
  member  = "serviceAccount:${google_service_account.test.email}"
}

resource "google_project_iam_member" "composer-service-agent-v2-ext" {
  project = "inlaid-ally-373906"
  role    = "roles/composer.ServiceAgentV2Ext"
  member  = "serviceAccount:${google_service_account.test.email}"
}`


However, while executing terraform apply, I am facing below err:


╷
│ Error: googleapi: Error 400: Composer API Service Agent service account (service-197833231297@cloudcomposer-accounts.iam.gserviceaccount.com) does not have required permissions set. Cloud Composer v2 API Service Agent Extension role might be missing. Please refer to https://cloud.google.com/composer/docs/composer-2/create-environments#grant-permissions and Composer Creation Troubleshooting pages to resolve this issue., failedPrecondition
│ 
│   with google_composer_environment.test,
│   on main.tf line 49, in resource "google_composer_environment" "test":
│   49: resource "google_composer_environment" "test" {
│

I referred to this document but didn't found a solution to the above issue. Any way to fix this error?

Tried to spin up GCP Cloud Composer using the below set of terraform script code base but facing the below error:

╷ │ Error: googleapi: Error 400: Composer API Service Agent service account (service-197833231297@cloudcomposer-accounts.iam.gserviceaccount.com) does not have required permissions set. Cloud Composer v2 API Service Agent Extension role might be missing. Please refer to ``https://cloud.google.com/composer/docs/composer-2/create-environments#grant-permissions`` and Composer Creation Troubleshooting pages to resolve this issue., failedPrecondition │ │ with google_composer_environment.test, │ on main.tf line 49, in resource "google_composer_environment" "test": │ 49: resource "google_composer_environment" "test" { │

Mazlum Tosun
  • 5,761
  • 1
  • 9
  • 23
dhruv
  • 21
  • 2

2 Answers2

0

When you create the Cloud Composer cluster for the first time, you have to give the roles/composer.ServiceAgentV2Ext to the Composer default Service Account, example with gcloud cli :

gcloud projects add-iam-policy-binding composer-env-account@{your_project}.iam.gserviceaccount.com \
    --member service-197833231297@cloudcomposer-accounts.iam.gserviceaccount.com \
    --role roles/composer.ServiceAgentV2Ext

Replace {your_project} by your project ID.

The default Composer service account for you is : service-197833231297@cloudcomposer-accounts.iam.gserviceaccount.com

The service account you gave to Composer will be used for the Airflow DAGs at runtime : composer-env-account

Mazlum Tosun
  • 5,761
  • 1
  • 9
  • 23
  • But where can I find service-197833231297@cloudcomposer-accounts.iam.gserviceaccount.com Service Account in the GCP Console to add the required role? – dhruv Jan 07 '23 at 01:29
  • In the IAM menu, you add principals with roles or add new roles to principals. You can also execute the gcloud command I shared in my answer, from your local Shell Session. – Mazlum Tosun Jan 07 '23 at 08:58
  • ```gcloud projects add-iam-policy-binding composer-env-account@{your_project}.iam.gserviceaccount.com \ --member service-197833231297@cloudcomposer-accounts.iam.gserviceaccount.com \ --role roles/composer.ServiceAgentV2Ext``` After running above command I was getting ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT: Request contains an invalid argument. so I ran ```gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ --member service-197833231297@cloudcomposer-accounts.iam.gserviceaccount.com \ --role roles/composer.ServiceAgentV2Ext``` same error – Abhishek Solanki Aug 20 '23 at 20:05
0

I had the same issue when spinning up cloud composer v2. I had to set an iam policy binding for service account used on composer environment config with role roles/composer.ServiceAgentV2Ext. In your similar case, the following command should work:

gcloud iam service-accounts add-iam-policy-binding\
composer-env-account@{YOUR_PROJECT_ID}.iam.gserviceaccount.com \
--member serviceAccount:service-197833231297@cloudcomposer-accounts.iam.gserviceaccount.com \
--role roles/composer.ServiceAgentV2Ext

resource "google_service_account_iam_member" "custom_service_account" {
   provider = google-beta
   service_account_id = "example-account@example-project.iam.gserviceaccount.com"
   role = "roles/composer.ServiceAgentV2Ext"
   member = "serviceAccount:service-00000000000@cloudcomposer-accounts.iam.gserviceaccount.com"
}

Relevant links:

  1. https://cloud.google.com/composer/docs/how-to/access-control#composer-sa
  2. Documentation for the code above: https://cloud.google.com/composer/docs/composer-2/create-environments#terraform_2:~:text=Project%20number.-,Example,-%3A
Vjosa Fusha
  • 1
  • 1