Google Cloud Composer "Error: Server Error"

Question

My Google Cloud Composer is not working since yesterday.

I can't access the airflow interface, returning:

Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.

A warning is showed in Google Cloud Compose Console:

UPDATE operation on this environment failed 9 hours ago with the following error message:
Composer Backend timed out. Currently running tasks are [stage: TP_DEPLOYMENT_UPDATING
response_timestamp {
  seconds: 1578617359
  nanos: 100000000
}
debug_info: "Deployment Manager experimental endpoint enabled"
].

There's no support in chat to Composer.

Answer 1

Did someone in your project recently try to update the Composer environment (e.g., install a new pypi package)? I think this article could be useful in this case, and also this one.

Composer has some undeclared dependencies and if they conflict with your pypi packages, this error will come up. The current workaround is to use either a virtualenvoperator or a kubernetes pod operator, so you never actually install your packages in the composer environment.

Also, Airflow web server logs are available in Stackdriver logging - it'll be useful to include them into your question with versions of Composer and Airflow.

EDIT You can try to upgrade (is a beta feature) your environment to the latest version following this instruction.