Hystrix Circuit breaker not opening the circuit

Question

I am implementing Circuit breaker using Hystrix in my Spring boot application, my code is something like below:

@service
public class MyServiceHandler {

    @HystrixCommand(fallbackMethod="fallback")
    
    public String callService() {
       // if(remote service is not reachable
       // throw ServiceException
    }
    
    public String fallback() {
       // return default response
    }
}

---

// In application.properties, I have below properties defined:
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=10000
hystrix.command.default.circuitBreaker.requestVolumeThreshold=3
hystrix.command.default.circuitBreaker.sleepWindowInMilliseconds=30000
hystrix.threadpool.default.coreSize=4
hystrix.threadpool.default.metrics.rollingStats.timeInMilliseconds=200000

I see that the fallback() is getting called with each failure of callService(). However, the circuit is not opening after 3 failures. After 3 failures, I was expecting that it will directly call fallback() and skip callService(). But this is not happening. Can someone advise what I am doing wrong here?

Thanks, B Jagan

Edited on 26th July to add more details below:

Below is the actual code. I played a bit further with this. I see that the Circuit opens as expected on repeated failured when I call the remote service directly in the RegistrationHystrix.registerSeller() method. But, when I wrap the remote service call within Spring retry template, it keeps going into fallback method, but circuit never opens.

@Service
public class RegistrationHystrix {
    Logger logger = LoggerFactory.getLogger(RegistrationHystrix.class);
    private RestTemplate restTemplate;
    private RetryTemplate retryTemplate;

    public RegistrationHystrix(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
        retryTemplate = new RetryTemplate();
        FixedBackOffPolicy fixedBackOffPolicy = new FixedBackOffPolicy();
        fixedBackOffPolicy.setBackOffPeriod(1000l);
        retryTemplate.setBackOffPolicy(fixedBackOffPolicy);
        SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
        retryPolicy.setMaxAttempts(3);
        retryTemplate.setRetryPolicy(retryPolicy);
    }

    @HystrixCommand(fallbackMethod = "fallbackForRegisterSeller", commandKey = "ordermanagement")
    public String registerSeller(SellerDto sellerDto)  throws Exception {
        String response = retryTemplate.execute(new RetryCallback<String, Exception>() {
                   @Override
                   public String doWithRetry(RetryContext context) {
                           logger.info(String.format("Retry count %d", context.getRetryCount()));
                          return restTemplate.postForObject("/addSeller", sellerDto, String.class);
                   }
       });
       return response;
    }

    public List<SellerDto> getSellersList() {
        return restTemplate.getForObject("/sellersList", List.class);
    }

    public String fallbackForRegisterSeller(SellerDto sellerDto, Throwable t) {
        logger.error("Inside fall back, cause - {}", t.toString());
        return "Inside fallback method. Some error occured while calling service for seller registration";
    }
}

Below is the service class which in turn calls the above Hystrix wrapped service. This class in turn is invoked by a controller.

@Service
public class RegistrationServiceImpl implements RegistrationService {

    Logger logger = LoggerFactory.getLogger(RegistrationServiceImpl.class);
    private RegistrationHystrix registrationHystrix;


    public RegistrationServiceImpl(RegistrationHystrix registrationHystrix) {
        this.registrationHystrix = registrationHystrix;
    }

    @Override
    public String registerSeller(SellerDto sellerDto)  throws Exception {
        long start = System.currentTimeMillis();
        String registerSeller = registrationHystrix.registerSeller(sellerDto);
        logger.info("add seller call returned in - {}", System.currentTimeMillis() - start);
        return registerSeller;

    }

So, I am trying to understand why the Circuit breaker is not working as expected when using it along with Spring RetryTemplate.

Answer 1

You should be using metrics.healthSnapshot.intervalInMilliseconds while testing. I guess you are executing all 3 request within default 500 ms and hence the circuit isn't getting open. You can either decrease this interval or you may put a sleep between the 3 requests.