How to run kubectl within a job in a namespace?

Question

Hi I saw this documentation where kubectl can run inside a pod in the default pod. Is it possible to run kubectl inside a Job resource in a specified namespace? Did not see any documentation or examples for the same..

When I tried adding serviceAccounts to the container i got the error:

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:my-namespace:internal-kubectl" cannot list resource "pods" in API group "" in the namespace "my-namespace"

This was when i tried sshing into the container and running the kubctl.

Editing the question.....

As I mentioned earlier, based on the documentation I had added the service Accounts, Below is the yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-kubectl  
  namespace: my-namespace   
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: modify-pods
  namespace: my-namespace
rules:
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
      - list
      - delete      
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: modify-pods-to-sa
  namespace: my-namespace
subjects:
  - kind: ServiceAccount
    name: internal-kubectl
roleRef:
  kind: Role
  name: modify-pods
  apiGroup: rbac.authorization.k8s.io      
---
apiVersion: batch/v1
kind: Job
metadata:
  name: testing-stuff
  namespace: my-namespace
spec:
  template:
    metadata:
      name: testing-stuff
    spec:
      serviceAccountName: internal-kubectl
      containers:
      - name: tester
        image: bitnami/kubectl
        command:
         - "bin/bash"
         - "-c"
         - "kubectl get pods"
      restartPolicy: Never

On running the job, I get the error:

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:my-namespace:internal-kubectl" cannot list resource "pods" in API group "" in the namespace "my-namespace"

Your pod is having permission for only default namespace. Please create service account with cluster admin permission then try. — Sachin Arote, Mar 27 '20 at 12:18
@VipinMenon If the provided answers did not solve your case, post your job operation and I can fully demonstrate how to correct it. — Will R.O.F., Mar 27 '20 at 13:31
@willrof I have modified the question to add the yaml file used — Vipin Menon, Mar 28 '20 at 06:17
@VipinMenon thanks for the provided info. I'll reproduce your scenario and until Monday I will provide you an answer! — Will R.O.F., Mar 28 '20 at 23:38
@VipinMenon I managed to validade your environment, please check my answer below with full reproduction and documentation references. Also consider upvoting if it is useful and accepting if it answers your question. — Will R.O.F., Mar 30 '20 at 10:39

Will R.O.F. · Accepted Answer · 2020-03-30T10:46:50.130

Is it possible to run kubectl inside a Job resource in a specified namespace? Did not see any documentation or examples for the same..

A Job creates one or more Pods and ensures that a specified number of them successfully terminate. It means the permission aspect is the same as in a normal pod, meaning that yes, it is possible to run kubectl inside a job resource.

TL;DR:

Your yaml file is correct, maybe there were something else in your cluster, I recommend deleting and recreating these resources and try again.
Also check the version of your Kubernetes installation and job image's kubectl version, if they are more than 1 minor-version apart, you may have unexpected incompatibilities

Security Considerations:

Your job role's scope is the best practice according to documentation (specific role, to specific user on specific namespace).
If you use a ClusterRoleBinding with the cluster-admin role it will work, but it's over permissioned, and not recommended since it's giving full admin control over the entire cluster.

Test Environment:

I deployed your config on a kubernetes 1.17.3 and run the job with bitnami/kubectl and bitnami/kubectl:1:17.3. It worked on both cases.
In order to avoid incompatibility, use the kubectl with matching version with your server.

Reproduction:

$ cat job-kubectl.yaml 
apiVersion: batch/v1
kind: Job
metadata:
  name: testing-stuff
  namespace: my-namespace
spec:
  template:
    metadata:
      name: testing-stuff
    spec:
      serviceAccountName: internal-kubectl
      containers:
      - name: tester
        image: bitnami/kubectl:1.17.3
        command:
         - "bin/bash"
         - "-c"
         - "kubectl get pods -n my-namespace"
      restartPolicy: Never 

$ cat job-svc-account.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-kubectl  
  namespace: my-namespace   
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: modify-pods
  namespace: my-namespace
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "delete"]      
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: modify-pods-to-sa
  namespace: my-namespace
subjects:
  - kind: ServiceAccount
    name: internal-kubectl
roleRef:
  kind: Role
  name: modify-pods
  apiGroup: rbac.authorization.k8s.io

I created two pods just to add output to the log of get pods.

$ kubectl run curl --image=radial/busyboxplus:curl -i --tty --namespace my-namespace
the pod is running
$ kubectl run ubuntu --generator=run-pod/v1 --image=ubuntu -n my-namespace
pod/ubuntu created

Then I apply the job, ServiceAccount, Role and RoleBinding

$ kubectl get pods -n my-namespace
NAME                    READY   STATUS      RESTARTS   AGE
curl-69c656fd45-l5x2s   1/1     Running     1          88s
testing-stuff-ddpvf     0/1     Completed   0          13s
ubuntu                  0/1     Completed   3          63s

Now let's check the testing-stuff pod log to see if it logged the command output:

$ kubectl logs testing-stuff-ddpvf -n my-namespace
NAME                    READY   STATUS    RESTARTS   AGE
curl-69c656fd45-l5x2s   1/1     Running   1          76s
testing-stuff-ddpvf     1/1     Running   0          1s
ubuntu                  1/1     Running   3          51s

As you can see, it has succeeded running the job with the custom ServiceAccount.

Let me know if you have further questions about this case.

score 0 · Answer 2 · answered Mar 27 '20 at 12:32

Create service account like this.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-kubectl

Create ClusterRoleBinding using this.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: modify-pods-to-sa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: internal-kubectl

Now create pod with same config that are given at Documentation.

There seems to have been some typos when I create it earlier the above manifest works — Vipin Menon, Mar 28 '20 at 08:42

Arghya Sadhu · Answer 3 · 2020-03-27T13:25:36.087

When you use kubectl from the pod for any operation such as getting pod or creating roles and role bindings it will use the default service account. This service account don't have permission to perform those operations by default. So you need to

create a service account, role and rolebinding using a more privileged account.You should have a kubeconfig file with admin privilege or admin like privilege. Use that kubeconfig file with kubectl from outside the pod to create the service account, role, rolebinding etc.
After that is done create pod by specifying that service account and you should be able perform operations which are defined in the role from within this pod using kubectl and the service account.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: internal-kubectl

How to run kubectl within a job in a namespace?

3 Answers3