Circuit breaker on http client can't close back after failure

Question

An example of circuit breaker over akka http client that looks totally simple doesn't work expected for me.

object HttpWithCircuitBreaker extends App {
  implicit val system = ActorSystem()
  implicit val materializer = ActorMaterializer()
  implicit val ec = system.dispatcher
  val breaker =
    new CircuitBreaker(
      system.scheduler,
      maxFailures = 2,
      callTimeout = 3.seconds,
      resetTimeout = 25.seconds)
      .onOpen(println("circuit breaker opened"))
      .onClose(println("circuit breaker closed"))
      .onHalfOpen(println("circuit breaker half-open"))
  while (true) {
    val futureResponse: Future[HttpResponse] = Http().singleRequest(HttpRequest(uri = "https://www.random.org/integers/?num=1&min=1&max=6&col=1&base=10&format=plain&rnd=new"))
    breaker.withCircuitBreaker(futureResponse).map(resp => resp.status match {
      case Success(_) =>
        resp.entity.dataBytes.runWith(Sink.ignore)
        println("http success")
      case _ =>
        resp.entity.dataBytes.runWith(Sink.ignore)
        println(s"http error ${resp.status.intValue()}")
    }).recover {
      case e@_ =>
        println(s"exception ${e.getMessage}")
    }
    Thread.sleep(1000)
  }
}

It starts perfectly fine fetching random numbers. It opens when disconnect from the net but never close back when I reconnect. As you can see from the log there is an attempt to recover but it fails with timeout:

http success
http success
http success
exception Tcp command [Connect(www.random.org:443,None,List(),Some(10 seconds),true)] failed
exception Circuit Breaker Timed out.
circuit breaker opened
exception Tcp command [Connect(www.random.org:443,None,List(),Some(10 seconds),true)] failed
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker Timed out.
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
circuit breaker half-open             <--- new http call should start here
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker Timed out.  <--- but it fails with timeout
circuit breaker opened
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast
exception Circuit Breaker is open; calls are failing fast

Gut feeling says that it must be something unconsumed http entity but still I'm not able to make it right.

Here is the sample on github to play with

score 3 · Accepted Answer · answered Feb 28 '17 at 12:13

3

For the circuit breaker to work properly, this val

val futureResponse: Future[HttpResponse] = Http().singleRequest(...)

should be a def

def futureResponse: Future[HttpResponse] = Http().singleRequest(...)

The circuit breaker needs to wrap the asynchronous call - and if you use a val, your asynchronous call will start outside the breaker.

Adding to that: refrain from using Thread.sleep in your code, try using - e.g. - Akka after.

answered Feb 28 '17 at 12:13

Stefano Bonetti

8,973
1
25
44

Thanks. Have finally figured that out that was about. – kharole Feb 28 '17 at 17:12

Circuit breaker on http client can't close back after failure

1 Answers1