How to combine CircuitBreaker with TimeLimiter and Bulkhead?

Question

I have a service that calls a dependency via REST. Service and dependency are part of a microservice architecture, so I'd like to use resilience patterns. My goals are:

Have a circuit-breaker to protect the dependency when it's struggling
Limit the time the call can run. The service has an SLA and has to answer in a certain time. On timeout we use the fallback value.
Limit the number of concurrent calls to the dependency. Usually the rate of calls is low and the responses are fast, but we want to protect the dependency against bursts and queue requests inside the service.

Below is my current code. It works, but ideally I'd like to use the TimeLimiter and Bulkhead classes as they seem to be built to work together.

How can I write this better?

@Component
class FooService(@Autowired val circuitBreakerRegistry: CircuitBreakerRegistry)
{
    ...

    // State machine to take load off the dependency when slow or unresponsive
    private val circuitBreaker = circuitBreakerRegistry
        .circuitBreaker("fooService")

    // Limit parallel requests to dependency
    private var semaphore = Semaphore(maxParallelRequests)

    // The protected function
    private suspend fun makeHttpCall(customerId: String): Boolean {
        val client = webClientProvider.getCachedWebClient(baseUrl)

        val response = client
            .head()
            .uri("/the/request/url")
            .awaitExchange()

        return when (val status = response.rawStatusCode()) {
            200 -> true
            204 -> false
            else -> throw Exception(
                "Foo service responded with invalid status code: $status"
            )
        }
    }

    // Main function
    suspend fun isFoo(someId: String): Boolean {
        try {
            return circuitBreaker.executeSuspendFunction {
                semaphore.withPermit {
                    try {
                        withTimeout(timeoutMs) {
                            makeHttpCall(someId)
                        }
                    } catch (e: TimeoutCancellationException) {
                        // This exception has to be converted because
                        // the circuit-breaker ignores CancellationException
                        throw Exception("Call to foo service timed out")
                    }
                }
            }
        } catch (e: CallNotPermittedException) {
            logger.error { "Call to foo blocked by circuit breaker" }
        } catch (e: Exception) {
            logger.error { "Exception while calling foo service: ${e.message}" }
        }

        // Fallback
        return true
    }
}

Ideally I'd like to write something like the docs describe for Flows:

// Main function
suspend fun isFoo(someId: String): Boolean {
    return monoOf(makeHttpCall(someId))
        .bulkhead(bulkhead)
        .timeLimiter(timeLimiter)
        .circuitBreaker(circuitBreaker)
}

How you have solved the problem? For Resilience4J Timelimiter require Schedular Service — Md Faraz, Feb 08 '22 at 15:29

score 0 · Answer 1 · answered Jun 08 '20 at 07:30

0

You could also use Resilience4j's Bulkhead instead of your own Semaphore and Resilience4j's TimeLimiter. You can stack you CircuitBreaker with bulkhead.executeSuspendFunction and timelimiter.executeSuspendFunction.

answered Jun 08 '20 at 07:30

Robert Winkler

1,734
9
8

Hi, thanks for your response. I'm not sure if the Bulkhead will perform, there is a warning about the performance here: https://resilience4j.readme.io/docs/getting-started-4#bulkhead With the `TimeLimiter`, do I still need to convert the exception? Ideally I'd like to not nest these calls but use something like a decorator. – fafl Jun 08 '20 at 08:27

How to combine CircuitBreaker with TimeLimiter and Bulkhead?

1 Answers1