5

After running for a couple of days Google Cloud Composer web UI returns the 502 Server Error indefinitely:

Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.

The only way to fix it is to recreate the Composer environment. Though after running for a couple of days the new environment crashes with the same error.

Image version: composer-1.4.0-airflow-1.10.0
Python version: 3

Anyone knows what's the root cause?

Meraj al Maksud
  • 1,528
  • 2
  • 22
  • 36
medvedev1088
  • 3,645
  • 24
  • 42

2 Answers2

1

I don't run Cloud Composer but I suspect that there's a case where the webserver has exited from all the web worker threads. This can sometimes happen when airflow has an extended timeout reading or writing to the database; either due to a held lock, or network connection issues. It probably is configured to restart if it fully exits, but there are some cases were the airflow webserver command will still hold on without exiting even though all web workers have exited.

Alternatively the 502 is about the identity provider implemented for GCP. If that's the case you might find you need to sign out of your Google login and use the sign in flow provided by Airflow (if it responds to a private browser session or a signed out session).

dlamblin
  • 43,965
  • 20
  • 101
  • 140
0

I was facing the same 502 error and it turned out to be an issue with the DAG itself. As mentioned:

"The web server parses the DAG definition files, and a 502 gateway timeout can occur if there are errors in the DAG."

Visible in Composer / Monitoring

monitoring

Web server was affected by an issue with the DAG itself. We solved it by deleting the recently added DAGs, after couple of minutes the Airflow UI was up.

remigiusz boguszewicz
  • 428
  • 4
  • 7