Loadbalancing fails when a server is down

Question

I have written a simple set of micro-services with the following architecture:

For all, I have added spring-boot-starter-actuator in order to add /health endpoint.

In Zuul/Ribbon configuration I have added :

zuul:
  ignoredServices: "*"
  routes:
    home-service:
      path: /service/**
      serviceId: home-service
      retryable: true

home-service:
  ribbon:
    listOfServers: localhost:8080,localhost:8081
    eureka.enabled: false
    ServerListRefreshInterval: 1

So that, each time client will call GET http://localhost:7070/service/home, loadbalancer will choose one of two HomeService which runs on 8080 or 8081 port and will call its endpoint /home.

But, when one of HomeService is shutdown, the loadbalancer does not seem to be aware (in spite of ServerListRefreshInterval configuration) and will fail with error=500 if it tries to call the shutdown instance.

How could I fix it?

I have posted the same question (as issue) in spring-cloud-netflix github repository : https://github.com/spring-cloud/spring-cloud-netflix/issues/1984 — OlivierTerrien, May 24 '17 at 20:47
You have static hosts so I doubt that will work. AFAIK that will only work if you use service discovery but as your routes are static it will operate on the same 2 instances always (regardless of status of the services). Zuul is for proxying not for service discovery. — M. Deinum, May 25 '17 at 05:42
Agree. The goal was not to use zuul as a discovery service but to use ribbon for loadbalancing in the same gateway than zuul. An that, without using any discovery service (like consul or eureka) — OlivierTerrien, May 25 '17 at 09:30

score 0 · Accepted Answer · answered May 25 '17 at 09:26

I have received and tested a solution from spring-cloud team.

Solution is here in github

To summarize:

I have added org.springframework.retry.spring-retry to my zuul classpath
I have added @EnableRetry to my zuul application
I have put the following properties in my zuul configuration

application.yml

server:
  port: ${PORT:7070}

spring:
  application:
    name: gateway

endpoints:
  health:
    enabled: true
    sensitive: true
  restart:
    enabled: true
  shutdown:
    enabled: true

zuul:
  ignoredServices: "*"
  routes:
    home-service:
      path: /service/**
      serviceId: home-service
      retryable: true
  retryable: true

home-service:
  ribbon:
    listOfServers: localhost:8080,localhost:8081
    eureka.enabled: false
    ServerListRefreshInterval: 100
    retryableStatusCodes: 500
    MaxAutoRetries: 2
    MaxAutoRetriesNextServer: 1
    OkToRetryOnAllOperations: true
    ReadTimeout: 10000
    ConnectTimeout: 10000
    EnablePrimeConnections: true

ribbon:
  eureka:
    enabled: false

hystrix:
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMilliseconds: 30000

jihor · Answer 2 · 2017-06-23T21:16:56.707

Debugging timeouts may be tricky, considering there are 3 levels of routing alone (Zuul→Hystrix→Ribbon), not including async execution layers and the retry engine. The following scheme is valid for Spring Cloud releases Camden.SR6 and newer (I've checked this on Dalston.SR1):

Zuul routes the request through RibbonRoutingFilter, which creates a Ribbon command with the request context. Ribbon command then creates a LoadBalancer command, which uses spring-retry for command execution, choosing retry policy for the RetryTemplate according to Zuul settings. @EnableRetry does nothing in this case, because this annotation enables wrapping methods with @Retryable annotation in retrying proxies.

This means, your command duration is limited to the lesser value of these two (see this post):

[HystrixTimeout], which is a timeout for invoked Hystrix command
[RibbonTimeout * MaxAutoRetries * MaxAutoRetriesNextServer] (retries kick in only if Zuul has them enabled in its configuration), where [RibbonTimeout = ConnectTimeout + ReadTimeout] on the http client.

For debugging, it's convenient to create a breakpoint in RetryableRibbonLoadBalancingHttpClient#executeWithRetry or RetryableRibbonLoadBalancingHttpClient#execute method. At this point, you have:

ContextAwareRequest instance (e.g. RibbonApacheHttpRequest or OkHttpRibbonRequest) with request context, which containes Zuul's retryable property;
LoadBalancedRetryPolicy intsance with load balancer context, which contains Ribbon's maxAutoRetries, maxAutoRetriesNextServer and okToRetryOnAllOperations properties;
RetryCallback instance with a requestConfig, which contains HttpClient's connectTimeout and socketTimeout properties;
RetryTemplate instance with chosen retry policy.

If the breakpoint is not hit, it means that org.springframework.cloud.netflix.ribbon.apache.RetryableRibbonLoadBalancingHttpClient bean was not instantiated. This happenes when the spring-retry library is not in the classpath.

Loadbalancing fails when a server is down

2 Answers2