1

I have a Kubernetes cluster setup with a cluster autoscaler, and I'm trying to get Istio behaving properly on this cluster.

In my setup, I'm stuck with a few things:

  • There are standalone pods that get scheduled on my nodes (not under any deployment/replicaset)
  • I have to use Istio CNI and can't use Istio init containers

The issue comes when the cluster autoscaler adds a new node when one of the aforementioned pods try to come up, but there aren't any resources available.

Istio CNI daemonset needs to finish bootstrapping the new node, but Kubernetes scheduler immediately schedules the pod when it sees that the node is in a ready state. There doesn't seem to be a way yet to stop kubernetes from marking the node ready until the daemonset finishes. This whole issue is covered in the Istio docs here.

For reasons I won't get into, I can't make the above pods a deployment/replicaset which probably rules out using Istio's recommended solution, since deleted pods won't spin up again.

The only solution I can think of right now is using a dynamic admission controller to mutate the pod spec and inject an init container that will watch the daemonset until it reaches the completed state.

So basically this leaves me with the following two questions:

  • Does the istio-validation container always get injected as the first init container? If so, this would make the above solution invalid.
  • This sounds like a bonkers way of handling this, and I can't help but believe there has got to be an easier way to do this. Any other ideas?

Any help would be much appreciated! Thanks in advance!

Cheers

Pranav Bhatt
  • 125
  • 1
  • 11

1 Answers1

0

The only other viable solution which I was able to implement is to create a new deployment that will watch the istio-cni daemonset pods.

If it finds these pods not in a running state, it will:

  1. Add taints (noschedule and noexecute) to the node the istio-cni pod is running on.
  2. Watch the pod and wait till it reaches running state
  3. Removes the taints from the node, thereby allowing the pods to finally spin up

Do ensure that you are aware of this potential side effect where if the node scales down with the custom added taints, the ASG will use this taint on any future nodes it spins up, preventing future pods from being scheduled on that NodeGroup.

Pranav Bhatt
  • 125
  • 1
  • 11