Trigger Kubernetes from from another kubernetes job

Question

I am running on kubernetes job (job-1) from base pod. It works for basic use case. For second use case, I want trigger another kubernetes job(job-2) from already running job: job-1. While running job-2 I get service account error as given below:

Error occurred while starting container for Prowler due to exception : Failure executing: POST at: https://172.20.0.1/apis/batch/v1/namespaces/my-namespace/jobs. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. jobs.batch is forbidden: User "system:serviceaccount:my-namespace:default" cannot create resource "jobs" in API group "batch" in the namespace "my-namespace".

I have created service account with required permissions as given below:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-sa-service-role-binding
subjects:
  - kind: ServiceAccount
    name: my-sa
    namespace: my-namespace
roleRef:
  kind: Role
  name: my-namespace
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-sa-service-role
rules:
  - apiGroups: [""]
    resources: ["secrets", "pods"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["batch", "extensions"]
    resources: ["jobs"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "list"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-sa

I am passing "my-sa" as service account name but still, it refers to default service account.

I am using fabric8io kubernetes client to trigger the job and below is my code:

          final Job job = new JobBuilder()
            .withApiVersion("batch/v1")
            .withNewMetadata()
            .withName("demo")
            .withLabels(Collections.singletonMap("label1", "maximum-length-of-63-characters"))
            .withAnnotations(Collections.singletonMap("annotation1", "some-annotation"))
            .endMetadata()
            .withNewSpec().withParallelism(1)
            .withNewTemplate()
            .withNewSpec().withServiceAccount("my-sa")
            .addNewContainer()
            .withName("prowler")
            .withImage("demo-image")
            .withEnv(env)
            .endContainer()
            .withRestartPolicy("Never")
            .endSpec()
            .endTemplate()
            .endSpec()
            .build();

Could you please set slf4j `loglevel=trace` and see if correct serviceaccount is sent in request payload by kubernetes client? — Rohan Kumar, Oct 20 '20 at 18:58
In my opinion it should work without an problems. KubernetesClient doesn't make any customization to provided resource and delegates everything to kubernetes via it's REST API — Rohan Kumar, Oct 20 '20 at 18:59

Abhishek Jaisingh · Answer 1 · 2020-10-20T08:46:51.253

If you see the error message in detail, you'll find that your client is not using the service account you created (my-sa). Its using the default service account in the namespace instead:

"system:serviceaccount:my-namespace:default" cannot create resource "jobs"

And it should be safe to assume, that the default service account will not be having the privileges to create jobs.

It should be worthwhile to look into the official documentation of fabric8io, to see how you can authenticate with a custom service-account. From what I could find in the docs, it should be mostly handled by mounting the secret, corresponding to the service-account into the pod, then configuring your application code or probably setting up an specific environment variable.

Trigger Kubernetes from from another kubernetes job

1 Answers1