0

I am managing Google Cloud through Terraform. Terraform Cloud authenticates to GCP, with a Service Account stored in the Terraform Cloud variable GOOGLE_CREDENTIALS.

When trying to create a BQ table from Google Sheet I am getting the error:

Error: googleapi: Error 403: Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials., accessDenied

And according to several SO questions, I can fix it adding the scope https://www.googleapis.com/auth/drive to the service account. However, I only find how to use the scopes in other use cases (i.e https://cloud.google.com/bigquery/docs/samples/bigquery-auth-drive-scope#bigquery_auth_drive_scope-python), where some kind of programming/CLI is involved, which is not the case of Terraform Cloud.

So how could I add that scope for Terraform Cloud authentication based on the GOOGLE_CREDENTIALS variable?

Javier Lopez Tomas
  • 2,072
  • 3
  • 19
  • 41
  • 2
    Terraform seems to use the application-default-credentials (ADCs) when no service account is supplied. Try below command to set ADCs with the needed scopes.: (The scopes are copied from the tf file in a previous message.) **gcloud auth application-default login --scopes=https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/bigquery** – Veera Nagireddy Apr 21 '23 at 08:37
  • 2
    Create an external table where you require **bigquery.tables.create** permission. Refer to [Required roles](https://cloud.google.com/bigquery/docs/external-data-drive#required_roles) – Veera Nagireddy Apr 21 '23 at 08:48
  • 1
    Hello @Javier Lopez Tomas, Feel free to update the status of the question. Let me know the answer below helps to resolve your issue? I am happy to help you if you have any further queries. – Veera Nagireddy Apr 24 '23 at 02:03
  • 1
    Hello @Javier Lopez Tomas, Please check my **EDIT 2** in the answer, which may help to resolve your issue. – Veera Nagireddy May 08 '23 at 07:13
  • Adding OAuth Scopes is not a solution and if possible, would not solve the problem. The problem **must** be solved by either: a) using a different identity with the correct IAM roles; b) adding an IAM role to the service account that authorized Terraform. This is an IAM permission problem and not an OAuth scope problem. – John Hanley Aug 17 '23 at 20:48

1 Answers1

1

Looks like a permissions issue when you are trying to create a BigQuery table with external data source (spreadsheets) from Terraform. Terraform seems to use the application-default-credentials (ADCs) when no service account is supplied.

If you have other BQ tables that are connected with spreadsheets and changing/adding those works out-of-the-box. Only for this specific resource after executing the below command to set ADCs with the needed scopes (The scopes are copied from the tf file):

gcloud auth application-default login --scopes=https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/bigquery

Also create an external table where you require bigquery.tables.create permission. Refer to Required roles, additionally, you can create an external table in BigQuery for a Drive data source (files in Drive) . The access token requires both "drive" and "bigquery" scope, otherwise the query will fail with a permission error.

EDIT1 : Refer to Using Terraform Cloud :

Place your credentials in a Terraform Cloud environment variable:

  1. Create an environment variable called GOOGLE_CREDENTIALS in your Terraform Cloud workspace.
  2. Remove the newline characters from your JSON key file and then paste the credentials into the environment variable value field. You can use the tr command to strip newline characters. cat key.json | tr -s '\n' ' '
  3. Mark the variable as Sensitive and click the Save variable.

All runs within the workspace will use the GOOGLE_CREDENTIALS variable to authenticate with Google Cloud Platform.

EDIT2:

scopes - (Optional) The list of OAuth 2.0 scopes requested when generating an access token using the service account key specified in credentials. Ensure that the scope of the VM/Cluster is set to or includes googleapis.com/auth/cloud-platform, By default, the following scopes are configured: googleapis.com/auth/cloud-platform and googleapis.com/auth/userinfo.email. Refer to Authentication Configuration.

References:

  1. 3nigm4's and vitorc.radi's answers on How to authenticate to gcp when using terraform cloud.
  2. Yaroslav Markovski's, HashiCorp Help Center article on How-to set up Google Cloud (GCP) credentials in Terraform Cloud
Veera Nagireddy
  • 1,656
  • 1
  • 3
  • 12
  • Hello Veera. Thanks for anssering. I am not using Terraform through a CLI, but with Terraform Cloud (as stated in the question, but i'll further clarity it in the body of it), so I have a service account stored in a variable in Terraform Cloud, and I don't use a CLI at all, so I cant add the --scopes argument – Javier Lopez Tomas Apr 24 '23 at 06:23
  • Hello @Javier Lopez Tomas, Please have a look at my edit in the answer. Also refer to similar [SO1](https://stackoverflow.com/questions/73000253) [SO2](https://stackoverflow.com/questions/59440068), which may help to resolve your issue. – Veera Nagireddy Apr 25 '23 at 02:44
  • Hello. I am already using the GOOGLE_CREDENTIALS. My question is how to add the scopes to that way of authenticating – Javier Lopez Tomas Apr 27 '23 at 22:25