1

In our project total 10 Glue jobs are running daily. I would like to build a dashboard to show last 7 days jobs status it means either succeeded or failure. Tried to achieve it in CloudWatch with metrics, but not able do it. Please give an idea to build this dashboard.

Billa
  • 59
  • 5
  • Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Mar 01 '22 at 06:05

1 Answers1

1

Probably a little late for the original questioner, but maybe helpful for others.

We had a similar task in our project. We have many jobs and need to monitor success and failure. In our experience, the built-in metrics aren't really reliable, nor do they really answer the question of whether a job was successful or not. But we found a good way for us by generating custom metrics in a generic way for all jobs. This also works for existing jobs afterwards without having to change the code. I wrote an article about it: https://medium.com/@ettefette/metrics-for-aws-glue-jobs-as-you-know-them-from-lambda-functions-e5e1873c615c

We have set cloudwatch alerts based on these metrics and we use the metrics in our grafana dashboard to monitor the glue jobs.

ettefette
  • 51
  • 3
  • Hi, what about https://docs.aws.amazon.com/glue/latest/ug/monitoring-chapter.html? I'm not sure it covers alerts though – pbsb Sep 21 '22 at 22:07
  • 1
    Hi pbsb, the documentary gives an overview of different ways of monitoring glue jobs. But I don't see a description of how an alarm could be set to "failure", for example. In some cases, using Glue Studio to get the status may be sufficient, but in our case we want to automatically generate an alert to the DevOps team. – ettefette Sep 23 '22 at 04:29