logstash alert after 1000 occurences

Question

I am trying to make Logstash to alert me only after it receives over 1000 items within 10 minutes. I need alerts in both Hipchat and PagerDuty.

My config seems reasonable, but does not work as expected.

filter {
    if my_filtering_conditional_that_is_100%_correct {
        throttle {
            before_count => 1000
            period => 600
            add_tag => ["PD"]
            key => "string"
        }
        clone {
            add_tag => ["Count"]
        }
    }
    if "Count" in [tags] {
        throttle {
            before_count => 1000
            period => 600
            add_tag => ["HC"]
            key => "string"
        }
    }
}

output {
    if "PD" in [tags] {
         pagerduty {
            event_type => trigger
            incident_key => "logstash/Logstash"
            service_key => Pagerduty_API_key
            workers => 1
            description => "Alert message"
        }
    }
    if "HC" in [tags] {
        hipchat {
            color => "random"
            from => "Logstash"
            format => "Alert message"
            room_id => "Room"
            token => "token"
        }
    }
}

score 4 · Accepted Answer · answered Feb 04 '16 at 16:42

4

You may have better success using the metrics filter.

filter {
  my_filtering_conditional_that_is_100%_correct {
    metrics {
      meter => [ "events" ]
      flush_interval => 600
      clear_interval => 600
      add_tag => "events"
    }
  }
}

output {
  if "events" in [tags] {
    if [events][count] > 1000 {
      # do things
    }
  }
}

answered Feb 04 '16 at 16:42

sysadmin1138

133,124
18
176
300

1

Didn't know about that filter. Very useful – alfredocambera Feb 05 '16 at 13:59
1

Worked. THX a lot! – Sart Feb 26 '16 at 17:04

score 1 · Answer 2 · answered Feb 04 '16 at 16:12

1

I think that your best option would be to use http://riemann.io/. It handle events "flows" and that kind of logic wouldn't be to difficult to represent there.

The example on the following link creates an alert when there are more that 5 events of a certain type:

(streams
  (where (<= 0 metric 5)
    (with :state "ok" index)
    (else
      (with :state "warning" index))))

http://riemann.io/howto.html#set-thresholds

Greetings,

answered Feb 04 '16 at 16:12

alfredocambera

446
2
12

Switching to another product is not an option. – Sart Feb 26 '16 at 17:03

logstash alert after 1000 occurences

2 Answers2