AWS EKS Service account cluster role bindings not working access Forbidden

Question

I`m facing issues with AWS EKS cluster roles looks like binding to service account it not working correctly.

Service Account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: operator
  namespace: operator

ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: operator
rules:
  - apiGroups: [""]
    resources: ["secrets", "configmaps", "services", "persistentvolumeclaims", "persistentvolumes", "pods"]
    verbs: ["create", "get", "list", "watch", "delete", "update", "patch"]

  - apiGroups: ["apps"]
    resources: ["statefulsets", "deployments"]
    verbs: ["create", "get", "list", "watch", "delete", "update", "patch"]

  - apiGroups: ["batch"]
    resources: ["cronjobs", "jobs"]
    verbs: ["create", "get", "list", "watch", "delete", "update", "patch"]

  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses", "volumesnapshotcontents", "volumesnapshots", "volumesnapshotcontents/status", "volumesnapshots/status"]
    verbs: ["create", "get", "list", "watch", "delete", "update", "patch"]

ClusterRoleBinding

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: operator
subjects:
- kind: ServiceAccount
  name: operator
  namespace: operator

Cluster Server Version: v1.17.6-eks-4e7f64

kubectl get deployments.apps -n operator operator -o yaml | grep service
  serviceAccount: operator
  serviceAccountName: operator

Python client tried with kubernetes==11.0.0 also kubernetes==9.0.0 and 10. Performing operations on pods and other base resources working fine. Code:

api_group = "snapshot.storage.k8s.io"
api_version = "v1beta1"
kind = "VolumeSnapshot"
namespace="default"
label_selector=""
api_response = client.CustomObjectsApi(api_client)\
     .list_namespaced_custom_object(api_group, api_version, namespace, kind,
                                    label_selector=label_selector)

Error logs:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py", line 1489, in list_namespaced_custom_object
    (data) = self.list_namespaced_custom_object_with_http_info(group, version, namespace, plural, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py", line 1595, in list_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 340, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 172, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 362, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 237, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '96d8e718-e6a8-45e5-a742-eb35dd65c8f8', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Wed, 29 Jul 2020 09:27:02 GMT', 'Content-Length': '411'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"VolumeSnapshot.snapshot.storage.k8s.io is forbidden: User \"system:serviceaccount:operator:operator\" cannot list resource \"VolumeSnapshot\" in API group \"snapshot.storage.k8s.io\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"snapshot.storage.k8s.io","kind":"VolumeSnapshot"},"code":403}

Also listing deployments failing:

config.load_incluster_config()
api_client = client.api_client.ApiClient()
deployments = client.AppsV1beta1Api(api_client)\
     .list_namespaced_deployment("default", label_selector="")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/apps_v1beta1_api.py", line 1843, in list_namespaced_deployment
    (data) = self.list_namespaced_deployment_with_http_info(namespace, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/apps_v1beta1_api.py", line 1931, in list_namespaced_deployment_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 340, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 172, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 362, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 237, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'cc91c8f8-8348-444c-ac61-aa6cd4d14e08', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 29 Jul 2020 09:26:54 GMT', 'Content-Length': '174'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested resource","reason":"NotFound","details":{},"code":404}

Any help highly appreciated, I'm really stuck.

your yamls mention namespace operator however the resource that you are trying to access is in default namespace. — Tarun Khosla, Jul 29 '20 at 11:36
ClusterRoles define permissions on namespaced resources and be granted across all namespaces according to https://kubernetes.io/docs/reference/access-authn-authz/rbac/ "ClusterRoles are cluster-scoped, you can also use them to grant access to: cluster-scoped resources (like nodes) non-resource endpoints (like /healthz) namespaced resources (like Pods), across all namespaces For example: you can use a ClusterRole to allow a particular user to run kubectl get pods --all-namespaces." — Philip Petrov, Jul 29 '20 at 11:43
Can you still try create a test volumesnapshot inside the "operator" namespace, just to verify that you are not running into some unexpected behaviour? I remember slightly that there are cases where ClusterRoleBinding are degraded to RoleBindings and thus get namespaced. After you know how it behaves in the same NS as the ServiceAccount, we can continue digging for the actuall issue. — Alexander Block, Jul 29 '20 at 12:26
Hi @AlexanderBlock, I tried to list it but now getting the 404 error as second snippet. HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested resource","reason":"NotFound","details":{},"code":404} — Philip Petrov, Jul 29 '20 at 13:51

score 0 · Accepted Answer · answered Nov 30 '20 at 12:22

0

I manage to find a solution the issue was that kind was wrong "VolumeSnapshot", I correct it to kind = "volumesnapshots" and worked. As s conclusion I can say always use a kind as it`s defined in the Cluster role. Lower cases!!!

answered Nov 30 '20 at 12:22

Philip Petrov

95
1
1
11

AWS EKS Service account cluster role bindings not working access Forbidden

1 Answers1