8

Just as the title says, Kubernetes's liveness probe always fails when pod is under heavy load(500+ concurrency, 100000 request), i gave more cpu & memory resources to the pod, but it does not work.

Is there any method to handle this situation ?

Thanks!

Williams Mao
  • 81
  • 1
  • 1
  • 4
  • scale up and use load balancer? – Ami Hollander Sep 02 '18 at 17:59
  • In production environment, when holiday comes, we have 40,000 concurrent requests, thus 80 pods are needed, but currently in production we use docker-compose based orchestration with only less than 10 containers @AmiHollander – Williams Mao Sep 02 '18 at 18:07
  • which server do you use? – Ami Hollander Sep 02 '18 at 18:12
  • tomcat, with 2 vcpu & 1G memory assigned – Williams Mao Sep 02 '18 at 18:20
  • I am worried about that when all pods are under heavy load, k8s will restart all pods, then the service will become unavailable – Williams Mao Sep 02 '18 at 18:32
  • Any logs you may want to provide? especially failures after you added cpu and memory resources. – Vit Sep 03 '18 at 10:51
  • I get the following event message for the pod under heavy load: Readiness probe failed: Get http://172.20.2.28:8080/render-web/index.html: net/http: request canceled (Client.Timeout exceeded while awaiting headers) It seems that the root cause is the failed readiness probe, after increasing the timeoutSeconds for readiness probe, everything works fine! I was thought that the readiness would not be performed once the pod is ready, so interesting! – Williams Mao Sep 03 '18 at 14:07

2 Answers2

2

As per @Williams Mao issue was resolved by increasing readinessProbe.timeoutSeconds.

If an application has huge load, the default timeoutSeconds: 1 may be not enough. Good to read: configure-liveness-readiness-probes

Vit
  • 7,740
  • 15
  • 40
0

Try changing probes by "kong health" command.

readinessProbe:                                                                                                                                                                                            
   exec:                                                                                                                                                                                                    
     command:                                                                                                                                                                                               
       - kong                                                                                                                                                                                               
       - health
Esteban Avendaño
  • 1