How can I monitor the Airflow web server when using Google Cloud Composer? If the web server goes down or crashes due to an error, I would like to receive an alert.
Asked
Active
Viewed 969 times
0
-
The web service for Composer is backed by the same technology behind [App Engine Flexible Environment](https://cloud.google.com/appengine/docs/flexible/). This means the web process should be fairly fault-tolerant and the service should be looking after the web server process. So, while this may be a common issue for non-managed Airflow, it should be less of a concern with Cloud Composer. Are there cases where the web service has failed in a Composer environment? – James Aug 28 '18 at 17:31
-
I don't have any such case, but i want to make sure that my service is running fine. Thanks for the feedback – Balaji Aug 29 '18 at 05:03
2 Answers
0
You can use Stackdriver Monitoring: https://cloud.google.com/composer/docs/how-to/managing/monitoring-environments. Alerts can also be set in Stackdriver.
kaxil
- 17,706
- 2
- 59
- 78
0
At this time, fine-grained metrics for the Airflow web server are not exported to Stackdriver, so it cannot be monitored like other resources in a Cloud Composer environment (such as the GKE cluster, GCE instances, etc). This is because the web server runs in a tenant project alongside your main project that most of your environment's resources live in.
However, web server logs for Airflow in Composer are now visible in Stackdriver as of March 11, 2019. That means for the time being, you can configure logs-based metrics for the web server log (matching on lines that contain Traceback, etc).
hexacyanide
- 88,222
- 31
- 159
- 162