system:node fails to get secrets from apiserver via curl

Question

I'm doing some POC for security research, trying to access namespace secrets directly from a worker node. I have a cluster on GKE running Kubernetes 1.20

I'm running the following command from a worker (none-master) node:

curl -v $APISERVER/api/v1/namespaces/default/pods/ \
  --cacert /etc/srv/kubernetes/pki/ca-certificates.crt \
  --cert /var/lib/kubelet/pki/kubelet-client.crt \
  --key /var/lib/kubelet/pki/kubelet-client.key

And it works fine.

However, trying to get secrets fails:

curl -v $APISERVER/api/v1/namespaces/default/secrets/ \
  --cacert /etc/srv/kubernetes/pki/ca-certificates.crt \
  --cert /var/lib/kubelet/pki/kubelet-client.crt \
  --key /var/lib/kubelet/pki/kubelet-client.key

{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "secrets is forbidden: User \"system:node:gke-XXX--YYY\" cannot list resource \"secrets\" in API group \"\" in the namespace \"pencil\": No Object name found",
"reason": "Forbidden",
"details": {
  "kind": "secrets"
},
"code": 403

Looking at the documentation, I see that kubelet running on node should be able to access secrets: https://kubernetes.io/docs/reference/access-authn-authz/node/

And from my understanding, the authorization is backed by the ClusterRole system:node. Looking at it I do see it has the role to get secrets:

% kubectl get clusterrole system:node -o json
{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "ClusterRole",
...
        {
            "apiGroups": [
                ""
            ],
            "resources": [
                "configmaps",
                "secrets"
            ],
            "verbs": [
                "get",
                "list",
                "watch"
            ]
        },
...
    ]
}

And some more relevant documentation for communication between kubelet and kube-apiserver: https://kubernetes.io/docs/concepts/architecture/control-plane-node-communication/#node-to-control-plane

score 3 · Answer 1 · answered Dec 22 '21 at 15:05

3

I think the certificates location you are giving is incorrect. I have tried the same on my plain kubernetes cluster with following certificates and it worked fine.

curl --cacert /etc/kubernetes/pki/ca.crt --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key $APISERVER/api/v1/namespaces/default/secrets

answered Dec 22 '21 at 15:05

confused genius

2,876
2
16
29

You sure you are seeing this on a worker node? On worker the path `/etc/kubernetes/pki` doesn't exist and I can't find any file named `apiserver-kubelet-client.crt` in the file system – Mugen Dec 23 '21 at 08:09
1

Verified that by design these certificates should not exist on non-master nodes. Node only has kubelet client certificate to communicate to kube-apiserver with. – Mugen Dec 24 '21 at 16:49

Mugen · Accepted Answer · 2021-12-24T16:59:52.640

After digging in the source code, I found the meaning of No Object name found error - secrets or configmaps must be named in order to be retrieved. As suggested by the documentation, they can be retrieved if mapped to the node in question by some pod.

Therefore assuming some secret server-password is used by some pod in my node, following command worked as expected:

curl -v $APISERVER/api/v1/namespaces/default/secrets/server-password \
  --cacert /etc/srv/kubernetes/pki/ca-certificates.crt \
  --cert /var/lib/kubelet/pki/kubelet-client.crt \
  --key /var/lib/kubelet/pki/kubelet-client.key

Also apparently kubectl can be used just as simply, its the one creating the certificates at /var/lib/kubelet/pki in the first place.

kubectl --kubeconfig /var/lib/kubelet/kubeconfig -n default get secret server-password -o json

score 0 · Answer 3 · answered Dec 23 '21 at 20:58

0

This is expected.

Because the access to secrets is permitted(normally) via service account. You need to find the service account token mounted on a pod running on the node. For this you may try to dig into "magical" /proc folder on the node. If you can access the service account token mounted on the pod and that service account has the permission to access secrets then only the secrets can be accessed from the nodes.

answered Dec 23 '21 at 20:58

Rajesh Dutta

229
1
5

This contradicts the documentation. Digging in deeper, I found that indeed you're limited by which secrets you can read, but read them you can, without anything except the node client certificate. – Mugen Dec 24 '21 at 16:48

system:node fails to get secrets from apiserver via curl

3 Answers3