0

I have a script in a VM that write data in a bucket in another project.

I want to schedule this script with Airflow but I have IAM access problem when the script need to write data:

AccessDeniedException: 403 148758369895-compute@developer.gserviceaccount.com does not have storage.objects.list access to ******

To launch the script I use the following command :

bash_command=' gcloud config set project project2 && gcloud compute --project "project1" ssh --zone "europe-west1-c" "VMname" --command="python script.py"',

If I want to launch the script with Google Cloud Shell, I need to use gcloud auth login but how can I do this with Airflow/Composer ??

I tried

bash_command='gcloud auth login && gcloud config set project project2 && gcloud compute --project "project1" ssh --zone "europe-west1-c" "VMname" --command="python script.py"',

without success

MohamedLEGH
  • 309
  • 1
  • 11

1 Answers1

0

The recommended way to share resources across GCP projects is to grant the service account associated with your Cloud Composer environment the minimum required IAM permissions in the target project.

In your case, you would grant the service account user 148758369895-compute@developer.gserviceaccount.com the objectViewer role on either the target project or the target bucket.

Be careful, though! The 148758369895-compute@developer.gserviceaccount.com appears to be the default Google Compute Engine service account. Granting additional permissions may also give other VMs in your Composer project the same permissions. To avoid this: create a custom service account and associate it with the Composer environment when you create the environment.

Tim Swast
  • 14,091
  • 4
  • 38
  • 61
  • Ok thanks. I created a custom service, associated with Composer environment on creation. After, I created and uploaded on the VM the key JSON file and when I connect on the VM, I use gcloud auth activate-service-account custom_service_account --key-file=key_file_path and after I can run my command without problems – MohamedLEGH Jun 19 '18 at 15:10
  • Depending on what else the VM is used for you might be able to eliminate the activate-service-account step by associating the service account with the VM when you create it. – Tim Swast Jun 19 '18 at 16:30