3

I'm playing with the bootiful-microservice project by Josh Long. (Brixton subproject)

On the reservation-service I have added a simple status method that can sleep a configurable amount of time to simulate load:

@RequestMapping(method = RequestMethod.GET, value = "/status")
public String status(){
    System.out.println("Checking status");
    try {
        Thread.sleep((long) (rand.nextDouble()*sleepTime));
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
    return "All is good";
}

The sleepTime variable is pulled from the Spring Config Server

On the reservation-client I have added an entry point in the gateway:

@FeignClient("reservation-service")
interface ReservationReader {

    @RequestMapping(method = RequestMethod.GET, value = "/reservations")
    Resources<Reservation> readReservations();

    @RequestMapping(method = RequestMethod.GET, value = "/status")
        String status();
    }
}

and I'm using an HystrixCommand

@HystrixCommand(fallbackMethod = "statusFallback")
@RequestMapping(method = RequestMethod.GET, value = "/status")
public String status(){
    return reader.status();
}

public String statusFallback(){
    return "Bad";
}

This all works well.

I set the sleeping time to 1500ms so that some request will be above the Hystrix default threshold (1000ms).

When I start hitting the API I get some failures due to timeout. If I hit long enough (50 times seems to work) the circuit breaker trigger and the circuit becomes open:

Hystrix Dashboard

My understanding is that as the downstream service becomes healthy again Hystrix will try to route 1 call and use it as a health check. If the call is successful circuit should be closed again.

However this is not happening here. The circuit will remain open even after changing the sleeping time to a smaller value (let's say 500ms). None of my calls are routed towards the reservation-services and the fallback is used on every call. The only way I can get the circuit to close again is to restart the reservation-client service.

Did I miss something? Is it an issue with Hystrix? Or with the Spring Integration?

UPDATE

I did further testing and I can confirm that the circuit will remain close forever, even after the sleeping has been reduced.

However if I use a route in the Zuul configuration I get the expected behaviour. The circuit closes itself if it sees a request that doesn't time out.

I have noticed another difference between forwarding by route compare to manually doing it in Spring. If I create a filter my /status/ call on the client does not trigger the filter. When I setup a route (eg. /foos/status => /status) it will trigger the filter and Hystrix behaves properly.

Is that a bug in Spring?

phoenix7360
  • 2,807
  • 6
  • 30
  • 41
  • 1
    Do you have the same problem even after a couple of minutes ? – reos May 10 '16 at 17:22
  • What does your reservation-client config look like. specifically: hystrix.command.default.circuitBreaker.sleepWindowInMilliseconds In Josh's example he has the config server pointed at a git repo on his local. – MarkOfHall May 11 '16 at 00:02
  • 1
    @reos: yes I do it stays like that for ever – phoenix7360 May 11 '16 at 07:42
  • @MarkOfHall: I have the same config as Josh on my local repo: hystrix.command.default.execution.isolation.strategy=SEMAPHORE – phoenix7360 May 11 '16 at 07:42
  • How are you refreshing the sleepTime in the reservation service? Are you certain Hystrix isn't half opening the circuit and timing out? – MarkOfHall May 12 '16 at 20:09
  • @MarkOfHall I did some more testing and found some interesting results. No Hystrix isn't half opening the circuit, even after I reduced the sleep time to 0. However I get the expected behaviour if I use a Zuul route rather than a manual forwarding in Spring (the circuit closes itself as soon as a request is healthy). I have also noticed that manual forwarding in Spring doesn't trigger filters like routes do. is that a bug? I will update my post – phoenix7360 May 13 '16 at 07:43

0 Answers0