1

I have an AKS cluster on Azure with virtual-nodes enabled (virtual-kubelet/azure-aci v1.3.2) and it works OK (a little flakey, but does mostly work). My problem is that as soon as I enable istio side car injection on any deployment I want to run on a virtual-node, the istio-proxy sidecar won't start, preventing the entire pod from starting up. I believe the problem is that v1:status.podIP is not yet supported in virtual-kubelet/azure-aci, and the istio sidecar spec makes use of it.

Has anyone had this problem? I can't find anything much about istio + virtual kubelet on the googlez.

I'm working around the problem for the time being by using PERMISSIVE mtls mode on the one dependency it relies on being able to make http calls to, and turning off side car injection via an annotation on the deployment. This is not ideal.

Thanks

AndyMoose
  • 425
  • 3
  • 11
  • Did you make any progress on this? Searching for "istio + virtual kubelet" on the googlez now leads here :) – roomaroo Mar 17 '21 at 09:51
  • Hey @roomaroo nope - didn't get anywhere at all with this. Frankly, been getting nada from azure support about this and other issues we've been experiencing on there. – AndyMoose Sep 15 '21 at 01:48
  • I stumbled upon the same issue today and was wondering if any of you guys able to make any progress on this? – ktumu Mar 13 '23 at 18:50
  • @ktumu nope - we gave up using ACI/Virtual-node about 6 months ago - between the not being able to use istio and other unsupported k8s features (graceful termination period) it just wasn't worth it. I haven't checked to see if these things have been resolved yet. – AndyMoose Mar 15 '23 at 00:32

1 Answers1

1

Edit: The Istio CNI plugin can be a workaround. Haven't tried it but the documentation says:

For application pods in the Istio service mesh, all traffic to/from the pods needs to go through the sidecar proxies (istio-proxy containers). This istio-cni Container Network Interface (CNI) plugin will set up the pods' networking to fulfill this requirement in place of the current Istio injected pod initContainers istio-init approach.

Reference: https://github.com/istio/istio/tree/master/cni

Istio uses an init container to get the istio container started. But Init containers are not yet supported in Virtual Nodes. That explains why the Istio is not getting started in your virtual nodes.

The microsoft documentation says:

Virtual Nodes functionality is heavily dependent on ACI's feature set. In addition to the quotas and limits for Azure Container Instances, the following scenarios are not yet supported with Virtual nodes:

  • Using service principal to pull ACR images. Workaround is to use Kubernetes secrets
  • Virtual Network Limitations including VNet peering, Kubernetes network policies, and outbound traffic to the internet with network security groups.
  • Init containers
  • Host aliases
  • Arguments for exec in ACI
  • DaemonSets will not deploy pods to the virtual nodes
  • Virtual nodes support scheduling Linux pods. You can manually install the open source Virtual Kubelet ACI provider to schedule Windows Server containers to ACI.
  • Virtual nodes require AKS clusters with Azure CNI networking.
  • Using api server authorized ip ranges for AKS.
  • Volume mounting Azure Files share support General-purpose V1. Follow the instructions for mounting a volume with Azure Files share
  • Using IPv6 is not supported.
iarunpaul
  • 117
  • 2
  • 13
  • CNI gets rid of istio-init container only and the sidecar envoy proxy would still get injected into app pods. – ktumu Mar 13 '23 at 18:53