Airflow 2.0 - Could not automatically determine credentials (GOOGLE_APPLICATION_CREDENTIALS)

Question

I have the following code from the example-dags at "https://github.com/apache/airflow/tree/master/airflow/providers/google/cloud/example_dags":

import os
from airflow import models
from airflow.providers.google.cloud.operators.bigquery import (
    BigQueryCreateEmptyDatasetOperator, BigQueryDeleteDatasetOperator,
)
from airflow.utils.dates import days_ago

PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "project_id")
DATASET_NAME = os.environ.get("GCP_BIGQUERY_DATASET_NAME", "dataset_id")

default_args = {"start_date": days_ago(1)}

with models.DAG( "test_dag" , default_args=default_args , schedule_interval=None , tags=["testing"] ) as dag:

create_dataset = BigQueryCreateEmptyDatasetOperator(
        task_id="create_dataset", dataset_id=DATASET_NAME
    )

    delete_dataset = BigQueryDeleteDatasetOperator(
        task_id="delete_dataset", dataset_id=DATASET_NAME, delete_contents=True
    )

    create_dataset >> delete_dataset

I just want to debug my Airflow connection with Google Cloud, creating a dataset and then removing it. I am currently getting this error:

Traceback (most recent call last): 
File "/home/airflow/.local/lib/python3.6/site-packages/google/cloud/_helpers.py", line 186, in _determine_default_project _, project = google.auth.default()
File "/home/airflow/.local/lib/python3.6/site-packages/google/auth/_default.py", line 321, in default raise exceptions.DefaultCredentialsError(_HELP_MESSAGE) google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

After getting this message, I tried to set an environment variable with an assigned path and to add it explicit in the code. None of this options worked. The problem is that any path seem to be unreachable "google.auth.exceptions.DefaultCredentialsError: File was not found"

You need to provide servicekey.json file and path as below : os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="/path/to/file.json" Di you just give the path without the file ? This should work actually. — Priya Agarwal, Jun 19 '20 at 07:49

score 1 · Answer 1 · answered Jul 02 '20 at 07:41

In order to omit this error it might be required to properly define Application Default credentials, obtaining valid authentication identity that has granted access to a specific resource across Google platform APIs.

In a common Airflow GCP authentication concept, I assume that you can consider, either:

Propagate environment variable GOOGLE_APPLICATION_CREDENTIALS approaching the path of a valid service account JSON private key file in the particular DAG code, as was proposed by @Priya Agarwal:
```
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="/FULLPATH/TO/JSON.key"
```

or

Create a separate Airflow connection through WEB UI, specifying in Keyfile Path field the path to the corresponded *.key file for the particular service account, defining then connection ID argument in DAG operator's parameters, as explained in this example.

Airflow 2.0 - Could not automatically determine credentials (GOOGLE_APPLICATION_CREDENTIALS)

1 Answers1

Linked