2

I am managing a Google Cloud Composer environment which runs Airflow for a data engineering team. I have recently been asked to troubleshoot one of the dags they run which is failing with this error : [12:41:18,119] {credentials_utils.py:23} WARNING - [redacted-name] connection ID not available, falling back to Google default credentials

The job is basically a data pipeline which reads from various sources and stores data into GBQ. The odd part is that they have a strictly similar Dag running for a different project and it works perfectly.

I have recreated the .json credentials for the service account behind the connection as well as the connection itself in Airflow. I have sanitized the code to see if there was any hidden spaces or so.

My knowledge of Airflow is limited and I have not been able to find any similar issue in my research, any one have encountered this before?

Notauser
  • 406
  • 2
  • 10
  • I think we are likely going to have to see more context of the puzzle. For example, what does the details of the task step in the DAG look like? What do the rest of the related messages look like? In your post, you redacted some text (good idea) ... but how does that text/value relate to other components ... is it named in the code in your DAG or as environment variables or something else. – Kolban Oct 04 '21 at 17:03
  • The service accounts are the same in both projects or different? – Raul Saucedo Oct 04 '21 at 22:53

1 Answers1

1

So the DE team came back to me saying it was actually a deployment issue where an internal module involved in service account authentication was being utilized inside another DAG running in stage environment, rendering it impossible to proceed to credential fetch from the connection ID.

Notauser
  • 406
  • 2
  • 10