0

we have an EKS cluster on 1.21.

There is an nginx-ingress-controller-default-ingress-controller deployed with a Classic Load Balancer.

Suddenly, its pods are crashing with following errors.

I0815 04:40:04.970835       8 flags.go:204] Watching for Ingress class: nginx
W0815 04:40:04.980149       8 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0815 04:40:04.980218       8 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0815 04:40:04.980255       8 client_config.go:548] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied
F0815 04:40:04.980417       8 main.go:272] Error while initiating a connection to the Kubernetes API server. This could mean the cluster is misconfigured (e.g. it has invalid API server certificates or Service Accounts configuration). Reason: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied

Below are the securityContext and VolumeMount of the pod.

 securityContext:
      allowPrivilegeEscalation: true
      capabilities:
        add:
        - NET_BIND_SERVICE
        drop:
        - ALL
      runAsUser: 101
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-k7n9n
      readOnly: true

I tried to change runAsUser : 0, but it throws back with message the pods ".." is invalid.

Can you please give me some directions on what can be wrong here and any possible solution?

Nisarg
  • 171
  • 8

1 Answers1

1

Try adding fsGroup. This will make serviceaccount directory readable by non-root user:

spec:
  template:
    spec:
      securityContext:
        fsGroup: 65534
Vasili Angapov
  • 8,061
  • 15
  • 31
  • Hello Vsaili, I added in pod yaml, but it throws error. # pods "nginx-ingress-controller-default-ingress-controller-79b9c4gb6kx" was not valid: # * : Invalid value: "The edited file failed validation": ValidationError(Pod.spec.containers[0].securityContext): unknown field "fsGroup" in io.k8s.api.core.v1.SecurityContext # I added under spec->containers->securityContext after runAsUser: 101. Pardon me if I am making changes at wrong place, since I am still learning this. – Nisarg Aug 15 '23 at 06:18
  • Also, I tried to add under spec->securityContext in next attemp, but it throws validation error. Currently it is set as securityContext: {} – Nisarg Aug 15 '23 at 06:22
  • You need to add `fsGroup` under `spec.template.spec.securityContext` – Vasili Angapov Aug 15 '23 at 06:23