ERROR kubebootstrap: WORKER PANICKED: ingresses.networking.k8s.io is forbidden: User cannot list resource "ingresses" in API group "networking.k8s.io"

Question

I am currently encountering an issue with my Amazon EKS cluster that hosts Lucidworks Fusion. To start with I had an Amazon EKS v1.18 cluster that I then upgraded to v1.19 which all went well. I also had ingress-nginx-3.7.1 running in my cluster that I then upgraded to ingress-nginx-4.0.19 everything still seemed to operate properly at that point. That said once I upgraded my cluster to v1.20, when draining my pods and restarting them in my new worker nodes all pods restarted properly except my Ambassador pod which is stuck in a CrashLoopBackOff state now. I looked into this pod's logs and found the following error:

2022/04/08 14:00:35 ERROR(s): kubebootstrap: WORKER PANICKED: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:sandbox1:sandbox1-ambassador" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "sandbox1"

Does anyone know what this means ? I believe this is telling me something related to permissions but I am not exactly sure. Also I would like to understand what caused this issue, was it upgrading my EKS cluster to v1.20 or was it upgrading ingress-nginx or something entirely different ? Lastly I'd like to understand how I can solve this issue and get this pod running properly again ? Any help is welcomed.

This is an issue with RBAC and it states very clearly that at this point service account `sandbox1-ambassador` does not have permissions to view `ingress` resource. Most correct solution to me is to upgrade `ambassador` itself via `helm` if it was installed this way. If it's not an option, then you need to find `role` related to `abmassador` in `sandbox1` namespace and correct it so it has `ingress` resource. [This is how it looks in general](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-example). — moonkotte, Apr 11 '22 at 08:04

score 6 · Accepted Answer · answered Apr 11 '22 at 15:17

After digging into this further I found that in order to fix this issue I needed to edit the sandbox1-ambassador role. So I executed the following command

kubectl edit role sandbox1-ambassador

and I appended the following to this configuration:

- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - list
  - watch
  - get

Once I saved these changes and closed the edit session I restarted the Ambassador pod and everything started working properly again.

ERROR kubebootstrap: WORKER PANICKED: ingresses.networking.k8s.io is forbidden: User cannot list resource "ingresses" in API group "networking.k8s.io"

1 Answers1