2

A company would like to scale their microservice application using namespaces, namespace represents a separate application per each client. The application consists of 22 microservices, the company has 80 clients, hence there will be 22 * 80 = 1760 pods in the cluster. The diagram is given below.

It was initially calculated that it would be enough to have 4 r4.xlarge nodes to keep proper performance. But the company has encountered a different kind of problem: the number of IP addresses per instance. For example, for 4 nodes of r4.xlarge type only: 4(nodes) * 4(network interfaces) * 15(Private IPv4 addresses) = 240 addresses available, but 1760 needed!

Increasing the number of nodes several times over budget. Are there any ways to avoid that IPs amount limitation? Or maybe you can propose another solutionenter image description here

GutsOut
  • 131
  • 1
  • 9
  • Kubernetes itself has a [maximum of 100 pods per node](https://kubernetes.io/docs/setup/best-practices/cluster-large/). You need more nodes if you want to run that many pods. – Chin Huang Mar 23 '21 at 17:28
  • Anyway, even so, can we avoid a limitation of 60 IPs to 100 IPs for pods in r4.xlarge instance? It will reduce amount of nodes significantly – GutsOut Mar 23 '21 at 17:49
  • 3
    Limitation is because you are using AWS VPC CN Plugin https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html. Using this plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network. As per Documentation, The Amazon VPC CNI plugin might not meet requirements for all use cases. Amazon EKS maintains a network of partners that offer alternative CNI solutions with commercial support options. For more information, see Alternate compatible CNI plugins. https://docs.aws.amazon.com/eks/latest/userguide/alternate-cni-plugins.html. – Sagar Velankar Mar 23 '21 at 17:57
  • @SagarVelankar You saved my day, thanks! Installing Calico helped https://goglides.io/replace-aws-vpc-cni-with-calico-on-aws-eks-cluster/732/ – GutsOut Mar 23 '21 at 18:55
  • 1
    Thanks for the link. Please note that you cannot use IP address targeting mode of AWS Load Balancer Controller. It will have to be Instance Targeting Mode. Please upvote my answer – Sagar Velankar Mar 23 '21 at 19:20

2 Answers2

2

This is something we intend to solve with an upcoming new version of the VPC CNI plugin. There is work being done to increase the number of VPC IP addresses available to pods on any given worker node. You can track this roadmap item here.

mreferre
  • 5,464
  • 3
  • 22
  • 29
1

Limitation is because you are using AWS VPC CN Plugin docs.aws.amazon.com/eks/latest/userguide/pod-networking.html. Using this plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network.

As per Documentation, The Amazon VPC CNI plugin might not meet requirements for all use cases. Amazon EKS maintains a network of partners that offer alternative CNI solutions with commercial support options. For more information, see Alternate compatible CNI plugins. https://docs.aws.amazon.com/eks/latest/userguide/alternate-cni-plugins.html

Please note that you cannot use IP address targeting mode of AWS Load Balancer Controller. It will have to be Instance Targeting Mode.

Sagar Velankar
  • 845
  • 5
  • 5