Circuit breaker for Scala and akka-http rest service

Question

I am currently implementing a circuit breaker with Akka-HTTP as follows:

 def sendMail(entity: MyEntity): ToResponseMarshallable = {

      Thread.sleep(5 * 1000)
      validateEntity(entity).map[ToResponseMarshallable] {
        case (body, subject) if !isEmpty(body, subject) => {
          val mailResponse = sendMail(body, subject)
          OK -> ProcessedEmailMessage(mailResponse)
        }
        case _ =>
          BadRequest -> s"error: for $entity".toJson
      }
    } catch {
      case e: DeserializationException => HttpResponse(BadRequest).withEntity(HttpEntity(s"error:${e.msg}").withContentType(ContentTypes.`application/json`))
    }
  }

  val maxFailures: Int = 2
  val callTimeout: FiniteDuration = 1 second
  val resetTimeout: FiniteDuration = 30 seconds

  def open: Unit = {
    logger.info("Circuit Breaker is open")
  }

  def close: Unit = {
    logger.info("Circuit Breaker is closed")
  }

  def halfopen: Unit = {
    logger.info("Circuit Breaker is half-open, next message goes through")

  private lazy val breaker = CircuitBreaker(
    system.scheduler,
    maxFailures,
    callTimeout,
    resetTimeout
  ).onOpen(open).onClose(close).onHalfOpen(halfopen)

  def routes: Route = {
    logRequestResult("email-service_aggregator_email") {
      pathPrefix("v1") {
        path("sendmail") {
          post {
            entity(as[EmailMessage]) { entity =>
              complete {
                breaker.withCircuitBreaker(Future(sendMail(entity)))
              }
            }
          }
        }
      }
    }
  }

My problem is that if i use breaker.withCircuitBreaker(Future(sendMail(entity))) the circuit breaker gets into an open state but the rest response returns There was an internal server error as a response

If instead I use breaker.withSyncCircuitBreaker(Future(sendMail(entity))) then the circuit breaker never goes in an open state but it returns the expected HttpResponse

Any thoughts on how can I solve this issue to trigger both the circuit breaker and also return a correct HTTP response ?

You need to use `onComplete` instead of `complete`. The `complete` directive expects the response to be ready to go immediately. On your case, `withCircuitBreaker` returns a `Future`, so `complete` won't be a valid option. The `onComplete` directive is setup to work with a `Future`, so it's a better fit here. Then inside the `onComplete` callback you can use `complete`. — cmbaxter, Mar 01 '16 at 14:15
@cmbaxter can you post an example, I'm failing to understand on how to bind/next the onComplete with complete — Lucian Enache, Mar 01 '16 at 14:23

Pim Verkerk · Accepted Answer · 2016-03-01T16:42:40.843

1

entity(as[EmailMessage]) { entity => ctx =>
  val withBreaker = breaker.withCircuitBreaker(Future(sendMail(entity)))
  val withErrorHandling = withBreaker.recover {
      case _: CircuitBreakerOpenException => 
        HttpResponse(TooManyRequests).withEntity("Server Busy")
  }
  ctx.complete(withErrorHandling)
}

edited Mar 01 '16 at 16:42

answered Mar 01 '16 at 14:34

Pim Verkerk

1,066
7
12

It is not working, I get the same "There was an internal server error" message. I have tried to do some adjustments and the sendMail() method to return directly a future, if I don't wrap that in the circuit breaker it works and the future is resolved. – Lucian Enache Mar 01 '16 at 14:42
The send mail takes at least 5 seconds while your call timeout is 1 second. Wouldn't this cause an exception? – Pim Verkerk Mar 01 '16 at 14:52
It should enter in an open state, which happens but no response is returned – Lucian Enache Mar 01 '16 at 14:55
I have removed the ```Thread.sleep()``` from the function call, and the response returns correctly. What I wanted now is just to return a HttpResponse from the ```onOpen``` function – Lucian Enache Mar 01 '16 at 15:08
You are right, I created a small app to test this and my first request works even if its takes longer then the timeout. Any subsequent request in the next 30 seconds will fail. – Pim Verkerk Mar 01 '16 at 15:09
There I was wondering when the CB changes state, is there any way to still return HttpResponses ? – Lucian Enache Mar 01 '16 at 16:22
that's awesome, it works for the second run where the open is on, but for the first run it works regularly only if I remove the ```Thread.sleep()```, if ```Thread.sleep()``` is present in the code, it would still throw an ```"There was an internal server error."``` and does not return the actual ```HttpResponse``` – Lucian Enache Mar 01 '16 at 17:08
You can add a `case ex => ...` to see if there is an other exception and handle it. – Pim Verkerk Mar 01 '16 at 17:13
on the withErrorHandling block? – Lucian Enache Mar 01 '16 at 17:17
In the `recover` part – Pim Verkerk Mar 01 '16 at 17:18
it was a TimeoutException that was not handled, thanks. – Lucian Enache Mar 01 '16 at 18:31

score 1 · Answer 2 · answered Mar 01 '16 at 19:47

I'm going to offer another possibile solution as I believe that onComplete is the mode idiomatic way to go when dealing with the result of a Future when completing a route:

entity(as[EmailMessage]) { entity =>
  val withBreaker = breaker.withCircuitBreaker(Future(sendMail(entity)))

  onComplete(withBreaker){
    case Success(trm) => 
      complete(trm)

    //Circuit breaker opened handling
    case Failure(ex:CircuitBreakerOpenException) => 
      complete(HttpResponse(TooManyRequests).withEntity("Server Busy"))

    //General exception handling
    case Failure(ex) =>
      complete(InternalServerError)
  }
}

Circuit breaker for Scala and akka-http rest service

2 Answers2