2

I have the following sensu check defined in puppet 

sensu::check { 'check_kubernetes_kubelet':
   interval    => 60,
   timeout     => 30,
   command     => 'check_http -H :::name::: -p 10248 -u /healthz -e "200 OK"',
   subscribers => ['kubernetes_minion'],
   contacts    => ['sensu_high'],
}

I want to add command retry to this check. This check should start alerting to its contacts only after N consecutive command executions return error.

In this example, say N is 2. Say a failure happens right after check_X. Then 60 seconds later check_X+1 will return error. After another 60 seconds check_X+2 will also fail. I want the alerting to start after check_X+2's error.

I think sensu filters may be useful. It looks like I need to create a filter like here. But then how do I associate that filter with this check? Do I need to define a custom handler too ?

Hakan Baba
  • 217
  • 2
  • 8

2 Answers2

1

You have to use handlers. When your check triggers and event, this gets to the handler or handlers you have defined for it (search for "handler" in checks). Then, each handler can have one or more filters associated.

From the doc:

When the Sensu server is processing an Event, it will check for the definition of a handler (or handlers). Prior to executing each Handler, the Sensu server will first apply any configured filter (or filters) for the Handler

Enrique Arriaga
  • 121
  • 2
  • Looking at the handler docs, what type of handler would you use? How would the filter for that look like? Looking at the enterprise edition, it looks like there is a handle_when [parameter](https://sensuapp.org/docs/1.0/enterprise/filters/handle-when.html#examples) to sensu::check. However I do not see it in the open source sensu::check [definition](https://github.com/sensu/sensu-puppet/blob/master/manifests/check.pp#L93) – Hakan Baba Oct 26 '17 at 05:29
  • The handler type depends on what you want to do with the data generated by the event. The two most used are "pipe", if you want to send it to a command (built-in or created by you), or "tcp/udp" to send the info to a socket. I'm not familiar with the other types... Regarding the filter, I think you need to use "occurences" https://sensuapp.org/docs/latest/reference/filters.html#example-handling-repeated-events – Enrique Arriaga Oct 26 '17 at 20:14
0

Occurences is what you need to use.
occurrences => 2,

tkjef
  • 389
  • 3
  • 8
  • 1
    Yes we end up using occurrences too. But we needed to put the occurrences field in the `custom` field. So it looked like `custom => { 'occurrences' => 3, },` – Hakan Baba May 03 '18 at 07:53