0

Bear with me while I try to figure out what is and isn't supported within autopilot... Not sure if this is related to autopilot, but any help is always appreciated.

I just spun up a GKE autopilot cluster, and while researching service meshes decided to try out cilium. To my surprise, cilium comes pre-installed on these nodes! I installed the cilium cli, and sure enough, it picked up the install. Unfortunately it shows several errors.

cilium status                                                                                                                                                                                           main 
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         8 errors
 \__/¯¯\__/    Operator:       disabled
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Containers:       cilium             Running: 7
                  cilium-operator    
Cluster Pods:     25/25 managed by Cilium
Image versions    cilium    gke.gcr.io/cilium/cilium:v1.11.1-gke3.8.1: 7
Errors            cilium    cilium         daemonsets.apps "cilium" not found
                  # several of these messages, 1 for each pod it would seem #
                  cilium    anetd-mvlgw    unable to retrieve cilium status: pods "anetd-mvlgw" is forbidden: User "brett.dudo@gmail.com" cannot create resource "pods/exec" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied

I'd love to turn on all the features that cilium has to offer, but I'm concerned I won't have the access necessary to do so. Is this a limitation within autopilot, or is there something I can do within IAM to address this?

Dudo
  • 4,002
  • 8
  • 32
  • 57
  • [Cilium comes pre installed on GKE Dataplane V2](https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine). kube-system is only meant for system pods and is managed by Google on Autopilot Mode. Try to create the clusters on [Standard Mode](https://cloud.google.com/kubernetes-engine/docs/how-to/dataplane-v2#gcloud) with Dataplane V2 enabled – Siegfred V. Oct 11 '22 at 01:17
  • I really like the idea behind autopilot... I'm going to try to live within its limitations for a bit. Thanks for the recommendation, though! – Dudo Oct 11 '22 at 16:06

1 Answers1

2

While GKE Dataplane V2 is based on Cilium, it is not actually vanilla OSS Cilium. GKE currently does not support many of the Cilium OSS features (although we are looking at adding support in the future). This is true for both Autopilot and Standard.

In terms of service mesh support on Autopilot, Anthos Service Mesh is fully supported. On GKE standard, Glue, Istion, Linkerd, etc all run perfectly fine.

Gari Singh
  • 11,418
  • 2
  • 18
  • 41
  • Thanks! I was looking at Anthos, but if feels... heavy. It seems to change the pricing of things as well? It seems like Anthos service mesh is a clone of sorts of istio? I really like how cilium is sidecar-less (since all traffic flows from node <> node, ultimately). Do you have a rough ETA on when more features will be enabled? – Dudo Oct 15 '22 at 20:45
  • 1
    No timeline on adding Cilium features yet. Not to push it, but there is standalone pricing for Anthos Service Mesh which does not require you to purchase the entire Anthos subscription. That being said, if you do want to play with a sidecar-less mesh on GKE, you might want to check out the Istio Ambient Mesh - https://istio.io/latest/blog/2022/introducing-ambient-mesh/ – Gari Singh Oct 15 '22 at 21:06
  • Regarding the unsupported features, I observe that the CiliumNetworkPolicy API is not available in GKE Dataplane V2. – Eron Wright Mar 08 '23 at 19:38