0

Is there anyone know how to reset the circuit breaker counters in istio v1.1.1? During performance testing(3 weeks ago), circuit breaker was triggered, there's thunder icon shown up in Kiali graph. After that, there's no traffic, the 'thunder' icon should disappear to my understanding. enter image description here

Cœur
  • 37,241
  • 25
  • 195
  • 267
Youfa Mao
  • 149
  • 1
  • 9

1 Answers1

0

I might not get it correctly, but to my understanding, there are 2 things.

  1. First minor one, the thunder icon in Kiali that you shared is only an indicator that there is Circuit Breaker applied to that resource. It won't actually go away unless the Circuit Breaker is deregistered. But you can choose what to display or not: enter image description here

Please check the Circuit Breaker specifications you have. From Kiali home, you can go to "Istio Config" and check the Circuit Breaker specifications. It can be inside the different "DestinationRule"(outlierDetection) or "DestinationPolicy"(CircuitBreaker)

  1. For the main thing:

how to reset the circuit breaker counters in istio v1.1.1?

Maybe you would want to assess your situation first with available information from Kiali.

  • From the Home page you can go -> Workloads ->
  • You will get detail page and can check tabs "Info", "Traffic", "Logs" and so on. You can check inbound and outbound traffic. If there is simply no request coming in, then, maybe need to check from the client of the service why the requests have stopped.

I feel you don't to do this. But unless you are completely aware and have assessed there's an issue, you can check individual Envoy Proxy stats using Kubectl pointing to your Kubernetes cluster. Istio Envoy Statistics reference.

kubectl -n {YOUR NAMESPACE} -it {THE POD NAME YOU WANT TO TEST} -c istio-proxy  -- sh -c 'curl localhost:15000/stats'

if you really want to reset:

kubectl -n {YOUR NAMESPACE} -it {THE POD NAME YOU WANT TO TEST} -c istio-proxy  -- sh -c 'curl -XPOST localhost:15000/reset_counters'
mOchi
  • 166
  • 1
  • 6
  • I've reset all the counters per your suggestion, but the kiali thunder icon still remains. For the circuit breaker config in DestinationRule, yes I've done this before, but the network flow is small and it shouldn't been triggered. – Youfa Mao Sep 05 '19 at 01:18
  • Sorry, I may not have understood correctly. Do you only want to remove the "thunder icon" in Kiali? That thunder icon indicates that there is a circuit breaker configuration. So, if you want to remove that thunder icon, is it correct for me to say that you want to remove the circuit breakers? If yes, you'll have to know where it is defined.If you are using v1alpha3 API, circuit breaker is defined in a Destination Rule (`kubectl -n {namespace} get destinationrules`). More details [here](https://istio.io/docs/concepts/traffic-management/#circuit-breakers). Then, modify the identified source. – mOchi Sep 09 '19 at 08:05
  • ooh... you're right - we've configured circuit breakers in destinationrule for those services. thanks! @mOchi – Youfa Mao Sep 11 '19 at 07:56