0

We are trying to create an Airflow Connection by setting an environment in Cloud Composer:

export AIRFLOW_CONN_GCP_CONN_ID=google_cloud_platform://?extra__google_cloud_platform__project=<project_id>&extra__google_cloud_platform__scope=https://www.googleapis.com/auth/cloud-platform&extra__google_cloud_platform__key_path=<path_to_json_file>

In logs we are getting an warning complaining that credentials are not set.

INFO - Getting connection using google.auth.default() since no key file is defined for hook.

When we looked into internals of code we found that if key_path and keyfile_data are not set then we get this error. But we are setting key_path as mentioned in command.

For internal code we looked into gcp_api_base_hook.py on GitHub.

hexacyanide
  • 88,222
  • 31
  • 159
  • 162
Sarang Shinde
  • 717
  • 3
  • 7
  • 24
  • Are you using Cloud Composer? Have you tried setting the environmental variable using the [gcloud](https://cloud.google.com/composer/docs/how-to/managing/environment-variables#adding_and_updating_environment_variables) command from cloud shell? `gcloud composer environments update ENVIRONMENT_NAME \ --location LOCATION \ --update-env-variables=KEY=VALUE` – ch_mike Nov 23 '18 at 18:53
  • We are able to create environment variables but problem is Airflow code failing to use it. – Sarang Shinde Nov 26 '18 at 04:13

1 Answers1

0

I am assuming you are using cloud composer.
You can save your service account credential file into airflow environment.
Goto Admin -- Connection select google_cloud_default and enter your service account details and the required scope you need to access.

Ashish Kumar
  • 542
  • 4
  • 20
  • Hi Ashish i am trying to create environment variables for connection. It is another way to use connections. Its working fine with airflow connection mechanism as you mentioned. But not working with another approach. – Sarang Shinde Nov 26 '18 at 04:12
  • @SarangShinde is this are you talking about? https://github.com/airflow-plugins/variable_connection_plugin/blob/master/operator/variable_connection_operator.py – Ashish Kumar Nov 26 '18 at 09:35