16

I'm using an AWS NLB to expose my Kubernetes pods to the internet. The NLB is currently using instance target types but I noticed there are also IP target types. What are the differences between an instance target type vs an IP target type from a practical point of view? When would you use one over the other?

AWS's documentation specifies some restrictions around using IP target types, namely they must be within certain CIDRs and constraints around maximums, but I'm trying to understand when you might want to use one over the other.

I don't know if it has any impact, but we've been having issues with our kubernetes rollingUpdate deployments where we're seeing downtime for the pods as they switch over (we have liveness and readiness checks there).

n00b
  • 5,843
  • 11
  • 52
  • 82

3 Answers3

11

The three key use-cases for using IP target type:

  • your target does not have to be an instance - anything with private IP address will work, including internal load balance, VPC private service, Fargate containers, databases, on-premise servers through VPN.
  • your target can be in different region, as long as you have cross-region peering between your VPCs
  • you have multiple network interfaces on your instance, so you can load distribute traffic between them, e.g. different applications on a single instance are bind to different interfaces. Each interface can be associated with different target group.

Instance target type is only limited instances. It should be your default choice when load balancing instances. For example, if you have instances in autoscaling group (ASG), the ASG can automatically register your instances with your load balancer. You can't do this for IP target types.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • So in my specific case where I'm using Kubernetes or EKS to be exact, should I be using an IP target as it's really pods that are the targets and we can have multiple pods on an instance? We're currently using `instance` targets and it seems to be fine as far as we're concerned, but perhaps there are some issues we might run into that we're just not aware of at the moment? – n00b Mar 09 '21 at 04:17
  • @n00b Not to sure about specifics of EKS and its load balancing capabilities. Sorry. – Marcin Mar 09 '21 at 04:19
2

Some additional considerations on this. Instance routing will forward to the NodePort of the service, which means the kubeproxy will receive the request and forward to a pod. In my configuration, which is default, it could forward to a pod on another node. This is generally not a big deal. If you're dealing with a lot of data, you might not want a lot of forwarding between nodes to be going on. Also, the node could be in another AZ (if using EKS and multiple AZs); and there would be an ingress and egress charge on the data crossing the AZ boundary.

Also, just to clarify something said above, if you are using IP routing in EKS; with the (latest) AWS Load Balancer Controller; it will keep the NLB target group updated with the POD IPs as you would expect. Or if doing instance routing it will keep the target group updated with the NodePort on each Node.

JeffG
  • 21
  • 1
1

In the context of Amazon Conainer Network Interface (CNI), Each Elastic Network Interface (ENI) on the node has a range of how many secondary IP addresses it can support. Pods are allocated IP addresses from this secondary Ip address range (that's the job of CNI).

When this range of secondary IP addresses is exhausted (maybe too many pods or too volatile pods), the CNI provision a new secondary ENI on the Node with a new range of secondary IP addresses for the pod (Depending on type of EC2 instace it has a limit to the number of ENIs that it can support).

As per NLB request routing.

If you specify targets using an instance ID, traffic is routed to instances using the primary private IP address that is specified in the primary network interface for the instance.

So now there is an impact on Pods density. And maybe that's happening with the rolling updates, that it exceeds the maximum limit.

Kapoor
  • 1,388
  • 11
  • 21