How can we visualize (via Dashboards) the Dataproc job status in Google Cloud Platform?
We want to check if jobs are running or not, in addition of their status like running, delay, blocked. On top of it we want to set alerting (Stackdriver Alerting) as well.
In this page, you have all the metrics available in Stackdriver
https://cloud.google.com/monitoring/api/metrics_gcp#gcp-dataproc
You could use cluster/job/submitted_count, cluster/job/failed_count and cluster/job/running_count to create the dashboard and metrics
Also, you could use cluster/job/completion_time to warn about long-running jobs and cluster/job/duration to check if jobs are enqueued in PENDING status for a long time.
cluster/job/completion_time is logged only after the job is completed. i.e. if the job takes 7 hours to complete, it is only registered at the 7th hour.
Similarly cluster/job/duration logs the time spent in each state only after the state is complete. Say if a job was in pending state for 1 hour, only at the 60th minute you would see this metric.
Dataproc has an open issue to introduce more metric that would help with this active alerting use case -> https://issuetracker.google.com/issues/211910984
