integrate eureka with kubernetes issue

Question

i have a (spring boot/spring cloud) application (micro-service 'MS' architecture) built with netflix tools and which i want to deploy it on kubernetes cluster (one master and 2 minions) to get advantage from its orchestration fact.

By the way, i created a kube-dns service on the cluster and i tried also to mount an eureka service (named eurekaservice) with 3 pods. From the other side, i run a micro-service with the next eureka configuration:

client:
  serviceUrl:
    defaultZone: http://eurekaservice:8761/eureka/

The good news is that every eureka pod on the cluster get notified about the new mounted MS instance. The bad news was that when the MS goes down, only one eureka pod get notified and the others no. Another thing, is that when i see the MS logfile, while mounted, it shows me the next errors:

Dec 01 09:01:54 ctc-cicd3 docker-current[1465]: 2016-12-01 06:01:54.469 ERROR 1 --- [nio-8761-exec-1] c.n.eureka.resources.StatusResource: Could not determine if the replica is available

Dec 01 09:01:54 ctc-cicd3 docker-current[1465]:
Dec 01 09:01:54 ctc-cicd3 docker-current[1465]: java.lang.NullPointerException: null
Dec 01 09:01:54 ctc-cicd3 docker-current[1465]: at com.netflix.eureka.resources.StatusResource.isReplicaAvailable(StatusResource.java:90)
Dec 01 09:01:54 ctc-cicd3 docker-current[1465]: at com.netflix.eureka.resources.StatusResource.getStatusInfo(StatusResource.java:70)
Dec 01 09:01:54 ctc-cicd3 docker-current[1465]: at org.springframework.cloud.netflix.eureka.server.EurekaController.status(EurekaController.java:63)
Dec 01 09:01:54 ctc-cicd3 docker-current[1465]: at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
Dec 01 09:01:54 ctc-cicd3 docker-current[1465]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 01 09:01:54 ctc-cicd3 docker-current[1465]: at java.lang.reflect.Method.invoke(Method.java:606)


Dec 01 09:02:16 ctc-cicd3 docker-current[1465]: 2016-12-01 06:02:16.918  WARN 1 --- [nio-8761-exec-8] com.netflix.eureka.InstanceRegistry      : DS: Registry: lease doesn't exist, registering resource: MS - gateway-bbn50:MS:8090
Dec 01 09:02:16 ctc-cicd3 docker-current[1465]: 2016-12-01 06:02:16.919  WARN 1 --- [nio-8761-exec-8] c.n.eureka.resources.InstanceResource    : Not Found (Renew): MS - gateway-bbn50:MS:8090
Dec 01 09:02:16 ctc-cicd3 docker-current[1465]: 2016-12-01 06:02:16.927  INFO 1 --- [nio-8761-exec-5] com.netflix.eureka.InstanceRegistry      : Registered instance id 12.16.64.2 with status UP
Dec 01 09:02:17 ctc-cicd3 docker-current[1465]: 2016-12-01 06:02:17.061  INFO 1 --- [io-8761-exec-10] com.netflix.eureka.InstanceRegistry      : Registered instance id 12.16.64.2 with status UP
Dec 01 09:02:46 ctc-cicd3 docker-current[1465]: 2016-12-01 06:02:46.932  WARN 1 --- [nio-8761-exec-9] com.netflix.eureka.InstanceRegistry      : DS: Registry: lease doesn't exist, registering resource: MS - gateway-bbn50:MS:8090

I think what was causing the problem is that replicas was unable to see each others.

how can i resolve this issue!!

score 1 · Answer 1 · answered Dec 04 '16 at 09:40

1

The problem here is that Eureka is a stateful app and you cannot scale it by just increasing the number of replicas.

See the Eureka "peer awereness" docs : http://cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.html

answered Dec 04 '16 at 09:40

Pierre Besson

740
3
6

Thanks for you reply @Pierre Besson. But i need to know how i need to proceed in this case ? – mootez Dec 04 '16 at 10:07
yes, you should. I suggest not to use pods, but configer single eureka services and configure them into a cluster. Kubernetes tries to abstract of this explicit configuration using internal loadbalancing and sd via DNS...while netflix does the opposite by offering a stack, which is very much agnostic to its underlying plattform – David Steiman Dec 04 '16 at 15:31
@David Steiman, the problem is that i have a already a java application running with ribbon, zuul and eureka, and i want to migrate above kubernetes to take advantage from its orchestration feature. i need to keep eureka running, otherwise, that impact of moving to DNS (which i don't know the size) may be very large. – mootez Dec 05 '16 at 06:58
One solution would be to remove eureka entirely and configure ribbon to use a static hostname. This way you would use Kubernetes load balancing rather than the normal client side load balancing of Ribbon. However this has some disadvantages as you would lose most advantages of Hystrix circuit breakers. The other solution is the one we advocate, to deploy a eureka server (and optionally replicas) to Kubernetes so would keep things as they are right now. – Pierre Besson Dec 05 '16 at 08:50
@Pierre Besson, thanksfor your response but what do you mean by "configure ribbon to use a static hostname" and how to do it ? i want this way (i won't use eureka i would like to work with kube-dns) – mootez Dec 06 '16 at 11:51

Pierre Besson · Accepted Answer · 2016-12-06T16:26:28.623

1

As it seems you want to replace Eureka discovery by Kubernetes discovery here is another answer (from http://cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.html) :

First disable eureka support in Ribbon:

ribbon:
  eureka:
   enabled: false

Then add a config property like this in your gateway for every microservice you want to access through your gateway (you could load those properties with a config server).

app1:   
  ribbon:
    listOfServers: app1-k8s-service-name
app2:   
  ribbon:
    listOfServers: app2-k8s-service-name

Then your gateway should be able correctly route calls to your microservices.

edited Dec 06 '16 at 16:26

answered Dec 06 '16 at 16:17

Pierre Besson

740
3
6

thanks a lot for you support. So what about the other micro-services (except gateway)? do i need to define config property like above ?because some micro-services needs to call each other internally to achieve some tasks. Also, what do you mean by 'app1', 'app2' ? In my case, i have one gateway micro-service with 4 other micro-services (training, message, settings and attachment). how may i proceed ? thanks again. – mootez Dec 07 '16 at 06:29
another thing, do i need to change something in Zuul config property ? @Pierre Besson – mootez Dec 07 '16 at 06:34
Yes replace app1, app2 by the applicationName of tour microservices. And if your microservices need to call themselves internally they would need the same kind ofconfig as the gateway. And normally no need to edit further Zuul config. – Pierre Besson Dec 07 '16 at 07:09
I needed to delete ServiceDiscoveryClient annotation from all micro-services as well as starter-eureka from pom.xml. i needed also to specify with seach app2-k8s-service-name its port. now things are working like a sharm. thank you so much @Pierre Besson – mootez Dec 08 '16 at 12:33

score 0 · Answer 3 · answered Apr 03 '20 at 12:24

Since you disabled eureka, only using zuul proxy :

server.ribbon.listOfServers=localhost:9090,localhost:9091,localhost:9092 server.ribbon.eureka.enabled=false

in case you are deploying on k8s (kubernetes), use the service name

ribbon: 
  eureka: 
    enabled: false

zuul:
  routes:
    organization-proxy:
      serviceId: organization-service
      path: /organization/**


organization-proxy:   
  ribbon:
    listOfServers: organization

integrate eureka with kubernetes issue

3 Answers3