Zuul retry configuration is not working with Eureka

Question

I have a scenario with Spring Boot Zuul as external Gateway and Eureka as Service Discovery, all this running in Kubernetes.

The thing is, I would like to guarantee my service's availability, so when of the instances of my service goes down, I expect Zuul to retry calling one of the other instances, through Eureka.

I tried doing this by following this Ryan Baxter's post. Plus, I tried to follow the tips from here.

The problem is that whatever I make, looks like Zuul is not retrying to make the call. When I remove one of my instances, it keeps returning me a Timeout for this instance, until Eureka addresses get synchronized.

My application.yaml looks like this:

spring:
  cloud:
    loadbalancer:
      retry:
        enabled: true

 zuul:
  stripPrefix: true
  ignoredServices: '*'
  routes:
    my-service:
      path: /my-service/**
      serviceId: my-service-api
  retryable: true

 my-service:
  ribbon:
    maxAutoRetries: 3
    MaxAutoRetriesNextServer: 3
    OkToRetryOnAllOperations: true
    ReadTimeout: 5000
    ConnectTimeout: 3000

My service is using Camden SR7 (I also tried SR6):

"org.springframework.cloud:spring-cloud-dependencies:Camden.SR7"

And also Spring-retry:

org.springframework.retry:spring-retry:1.1.5.RELEASE

My application class looks like this:

@SpringBootApplication
@EnableEurekaClient
@EnableZuulProxy
@EnableRetry
public class MyZuulApplication

EDIT:

Making a get through Postman, it brings

{
    "timestamp": 1497959364819,
    "status": 500,
    "error": "Internal Server Error",
    "exception": "com.netflix.zuul.exception.ZuulException",
    "message": "TIMEOUT"
}.

Taking a look at the Zuul logs, it printed {"level":"WARN","logger_name":"org.springframework.cloud.netflix.zuul.filters.post.SendErrorFilter","appName":...,"message":"Error during filtering","stack_trace":"com.netflix.zuul.exception.ZuulException: Forwarding error [... Stack Trace ...] Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: my-service-api timed-out and no fallback available [... Stack Trace ...] Caused by: java.util.concurrent.TimeoutException: null

Another interesting log that I found:

{"level":"INFO" [...] current list of Servers=[ip_address1:port, ip_address2:port, ip_address3:port],Load balancer stats=Zone stats: {defaultzone=[Zone:[ ... ];    Instance count:3;   Active connections count: 0;    Circuit breaker tripped count: 0;   Active connections per server: 0.0;]
},Server stats: [[Server:ip_address1:port;  [ ... ] Total Requests:0;   Successive connection failure:0;    Total blackout seconds:0;   [ ... ]
, [Server:ip_address2:port; [ ... ] Total Requests:0;   Successive connection failure:0;    Total blackout seconds:0;   [ ... ]
, [Server:ip_address3:port; [ ... ] Total Requests:0;   Successive connection failure:0;    Total blackout seconds:0;   [ ... ]

What is the exact exception you got ? – yongsung.yoon Jun 20 '17 at 02:18 — yongsung.yoon, Jun 20 '17 at 02:18
I edited the original post with some more information – Flavio Emilio Borges Jun 20 '17 at 12:15 — Flavio Emilio Borges, Jun 20 '17 at 12:15

score 0 · Accepted Answer · answered Jun 20 '17 at 12:04

0

The problem seems to be caused by Hystrix timeout. The default timeout of HystrixCommand is 1000ms and it's not enough for ribbon to retry http request. Try to increase the timeout of hystrix like followings.

hystrix:
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMilliseconds: 20000

It will increase whole hystrix command's timeout to 20 seconds. If it works, please adjust above value for your environment. You are using quite big timeout value for read and connect timeout. Therefore, you need to adjust those values with hystrix timeout if needed.

answered Jun 20 '17 at 12:04

yongsung.yoon

5,489
28
32

Thanks for helping. I will try it later on and then I post here. – Flavio Emilio Borges Jun 20 '17 at 12:16
I tested and it worked like a charm. I'll adjust the timeouts to something that makes more sense to me. Thank you! – Flavio Emilio Borges Jun 20 '17 at 16:33

Zuul retry configuration is not working with Eureka

1 Answers1