6

I created some Dashboards in Grafana to monitor the logs of the Kubernetes Pods. Below is a screenshot taken from a part of the Dashboard. So it basically shows "Fail" logs in red color as below. But I need something alerting to be done whenever if a "Fail" log occured.

enter image description here

enter image description here

It doesn't show me to add "Alerts" under this Dashboard. I need a solution that Alerts Fail logs. Can someone tell me how can I accomplish this task?

Container-Man
  • 434
  • 1
  • 6
  • 17

1 Answers1

8

Here's an explainer video - while in that video they use Loki as prometheus type data source, it works just as well when using loki type.

TL;DW

  1. Open the alerting section in Grafana and create an alert
  2. Define a rule - note that you need to use a metric query (see docs), e.g.:
count_over_time(
  {app="bot"}[15m]
    |= "ERROR"  
)

This rule will return a graph of the count of log lines containing ERROR within 15 minutes

If you need the total count, wrap it with sum(...)

TeNNoX
  • 1,899
  • 3
  • 16
  • 27