Scalability of kube-proxy

Question

I have encountered a scalability problem when trying out the kubernetes cluster. To simplify the topology in my test machine, NodePort type is used to expose the individual service externally. The baremetal to host the node and master is a RHEL 7 with 24 CPUs and 32G RAM; I don't yet have a dedicated load balancer, or a cloud provider like infrastructure. A snippet of the service definition looks just like below

    "spec": {       
         "ports": [{
             "port": 10443,             
             "targetPort": 10443,               
             "protocol": "TCP",                                
             "nodePort": 30443
    } ],
   "type": "NodePort",

Through this way the application can be accessible via https://[node_machine]:30443/[a_service]

Such service is only backed by one Pod. Ideally I would want to have several services deployed on the same node (but using different NodePort's), and and running concurrently.

Things were working well until it became evident that for a similar workload, increasing the number of services deployed (therefore backend pods as well) makes the applications degrade in performance. Surprisingly, when breaking down the service loading time, I noticed there's dramatic degradation in 'Connection Time' which seems to indicate there is a slowdown somewhere in the 'network' layer. Please note that the load isn't high enough to drive much of the CPU on the node yet. I read about the shortcomings in the doc, but not sure if what I hit is exactly the limitation of the kube-proxy/Service described there.

The questions are:

Is there any suggestion on how to make it more scalable? I.e. to be able to support more services/Pods without scarifying the applications' performance? The NodePort type is the easiest way to setup the 'public' address for our services but is there any limitation for scalability or performance if all services and Pods are setting up this way?
Would there be any difference if we change the type to LoadBalancer? "type": "LoadBalancer"
Further more, is there a benefit to have a dedicated LoadBalancer or reverse proxy to improve the scalability, e.g. HAProxy or alike, that routes traffic from external to the backend Pods (or Services)? I noticed there's some work done for Nginx darkgaro/kubernetes-reverseproxy - unfortunately the doc seems incomplete and there's no concrete example. In some of the other threads folks talked about Vulcan - is it the recommended LB tool for kubernetes?

Your recommendation and help are highly appreciated!

Welcome to Stack Overflow! I noticed you had some trouble formatting your question in the style you wished. I corrected it for the markdown syntax used here. You may find it helpful to read a bit more about the [layout formatting used](http://stackoverflow.com/help/formatting). — Brian Tompsett - 汤莱恩, Aug 20 '15 at 16:27

score 1 · Answer 1 · answered Sep 01 '15 at 12:58

Hello am kinda new to kubernetes but I have similar questions and concerns. Will try to answer some of them or redirect you to the relevant sections of the user guide.

In case you are deploying Kubernetes on a non cloud enabled providers like for example vagrant /local, etc then some features are not currently offered or automated by the platform for u.

One of those things is the 'LoadBalancer' type of Service. The automatic provision and assignment of a PUBLIC IP to the service (acting a L.B) happens currently only in platforms like Google Container engine.

See issue here and here.

The official documentation states

On cloud providers which support external load balancers, setting the type field to "LoadBalancer" will provision a load balancer for your Service.

Currently an alternative is being developed and documented, see here using HAProxy.

Maybe in the near future, kubernetes will eventually support that kind of feature in all the available platforms that can be deployed and operate, so always check their updated features.

What you are referring as performance degrade is most probably due to the fact, PublicIP (NodePort from version 1.0 and onwards) feature is working. Meaning that with the use of NodePort service type, kubernetes assigns a port on ALL nodes of the cluster for this kind of service. Then the kube-proxy intercepts the calls to this ports to the actual service etc.

An example on using HaProxy trying to solve the very same problem can be found here.

Hope that helped a bit.

score 0 · Answer 2 · answered Nov 05 '15 at 08:58

I'm facing same problems. It seems that internal kube-proxy is not intended to be external load balancer. More specifically, we wanted to setup some timeout on kube-proxy or do re-tries etc.

I've found this article which describes similar issues. He recommends to look at vulcan as it internally uses etcd and probably the direction of this project is to provide fully featured LB for k8s in the future.

Scalability of kube-proxy

2 Answers2