forbidden: User , "code": 403 error while accessing kube-apiserver using curl

Question

Kubernetes version : v1.19.0

I have created a user and performed clusterrolebinding with a role cluster-admin.

[root@project1-master ~]# kubectl describe clusterrole cluster-admin
Name:         cluster-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  *.*        []                 []              [*]
             [*]                []              [*]

[root@project1-master ~]# kubectl describe clusterrolebinding sajeesh  cluster-admin
Name:         sajeesh
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind  Name     Namespace
  ----  ----     ---------
  User  sajeesh


Name:         cluster-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind   Name            Namespace
  ----   ----            ---------
  Group  system:masters

I am able to run kubectl with this useraccount and get pods information :

[root@project1-master ~]# kubectl get pods --as sajeesh
NAME    READY   STATUS    RESTARTS   AGE
busyb   1/1     Running   3          21h

But when i try to access kube-apiserver using curl it show forbidden error as following :

[root@project1-master ~]# curl --cacert /etc/kubernetes/pki/ca.crt --cert sajeesh.crt --key sajeesh.key https://$IP:6443/api/v1/namespaces/default/pods/busyb
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "pods \"busyb\" is forbidden: User \"system:anonymous\" cannot get resource \"pods\" in API group \"\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {
    "name": "busyb",
    "kind": "pods"
  },
  "code": 403

I have re-verified the cacert , cert & key i am providing with that user account .They are correct.

Any suggestions why this is happening and how to fix it.

Arghya Sadhu · Answer 1 · 2020-09-12T15:49:59.457

1

The clusterrolebinding sajeesh has name sajeesh and Group system:masters.Hence the certificate need to have sajeesh as Common Name(CN) and system:masters as Organization(O). When the request goes to kubernetes API server it will inspect the certificate and match CN with name and O with Group in clusterrolebinding and if it does not match you get 403 Forbidden error.

You can verify above by running below command

openssl x509 -in sajeesh.crt -text -noout

edited Sep 12 '20 at 15:49

answered Sep 12 '20 at 14:59

Arghya Sadhu

41,002
9
78
107

Yes CN is sajeesh , But O is different becuase i want this user to be a part of different group named group1. Signature Algorithm: sha256WithRSAEncryption Issuer: CN=kubernetes Validity Not Before: Sep 12 07:44:59 2020 GMT Not After : Sep 12 07:44:59 2021 GMT Subject: CN=sajeesh, O=group1 – confused genius Sep 12 '20 at 15:00
As I said `O` need to have `system:masters` because that's what you have in the clusterrolebinding..otherwise edit or create a new clusterrolebinding with `Group` `group1` – Arghya Sadhu Sep 12 '20 at 15:05
if you see the output of 'kubectl describe clusterrolebinding sajeesh cluster-admin' in my question , Binding is done to both user name sajeesh and also to group system:masters.so ideally it should provided access to this user as well and one more reason to support this thought is "kubectl get pods --as sajeesh" is working fine . – confused genius Sep 12 '20 at 15:11
I have recreated the user(csr&certificate) Mentioning O as system:masters Issuer: CN=kubernetes Validity Not Before: Sep 12 15:50:00 2020 GMT Not After : Sep 12 15:50:00 2021 GMT Subject: CN=sajeesh, O=system:masters It is still showing same error when i access it using curl . "message": "pods \"busyb\" is forbidden: User \"system:anonymous\" cannot get resource \"pods\" in API group \"\" in the namespace \"default\"" – confused genius Sep 12 '20 at 15:54
1

Can you check the client certificate that you have in kubeconfig. Compare that with the generated certificate and see if there is any difference other than the CN – Arghya Sadhu Sep 12 '20 at 15:56

confused genius · Accepted Answer · 2020-10-05T20:26:46.150

finally managed to findout the problem.it is not related to kubernetes but with curl command i am using .

curl --cacert /etc/kubernetes/pki/ca.crt --cert sajeesh.crt --key sajeesh.key https://$IP:6443/api/v1/namespaces/default/pods/busyb

when i used -v switch along with command , It showed :

* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/kubernetes/pki/ca.crt
  CApath: none
* warning: certificate file name "sajeesh.crt" handled as nickname; please use "./sajeesh.crt" to force file name
* NSS: client certificate not found: newsajeesh.crt
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

So basically it was looking for absolute path as input for arguments --cert & --key

curl --cacert /etc/kubernetes/pki/ca.crt --cert ./sajeesh.crt --key ./sajeesh.key https://$IP:6443/api/v1/namespaces/default/pods/busyb

after giving the absolute path it worked fine and i am able to get the output.

forbidden: User , "code": 403 error while accessing kube-apiserver using curl

2 Answers2