Schedule a job with replicated containers on different hosts

Question

I am trying to run a Kubernetes job that will deploy the same pod on each host and run the same command in each of the pods (identical execution). There are 5 workers in the cluster worker01, worker02, ..., worker05. This definition schedules one pod on a random host:

apiVersion: batch/v1
kind: Job
metadata:
  name: hdd-integrity
  labels:
    job: hdd-integrity
spec:
  template:
    spec:
      containers:
        - name: hdd-integrity
          image: some-image
          imagePullPolicy: IfNotPresent
          command: ["python", "check.py"]
          args:
            - "--threads=88"
            - "--md5=2a3f98b6eb50326cf69257c5c5fc7e35"
            - "--dir=/mnt"
          volumeMounts:
            - name: store
              mountPath: /mnt
      restartPolicy: Never
      volumes:
        - name: store
          persistentVolumeClaim:
            claimName: subgrp1-subvol1-pvc
            readOnly: false
  backoffLimit: 1

All pods should mount to the same pvc. Not sure what is the best way to achieve this. Daemonsets will not work because they do not provide restartPolicy: Never

score 1 · Answer 1 · answered Jan 28 '21 at 06:34

A Job will be scheduled to a random node and that's by design. To run a workload in each node use daemonset.

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

score 1 · Answer 2 · answered Jan 28 '21 at 06:36

If you want to schedule a pod per node for a specific application, the right way to do so in the Kubernetes is using the Daemonset.

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

Further, if you want to schedule each pod in a different node, you can look at pod anti-affinity concepts.

score 0 · Answer 3 · answered Jan 29 '21 at 03:38

DaemonSets do not have restartPolicy: Never however this job needs to only run once. So a simple solution will be to use jinja2 templates:

{%- set workers = ["worker01", "worker02", "worker03", "worker04", "worker05"]
%}
{%- for w in workers %}
{%- set worker = w %}
---
apiVersion: batch/v1
kind: Job
metadata:
  name: hdd-integrity-job-{{ worker }}
  labels:
    job: hdd-integrity
    tag: "0.0.6"
spec:
  template:
    spec:
      containers:
        - name: hdd-integrity
          image: my-image
          imagePullPolicy: IfNotPresent
          command: ["/bin/bash", "-c", "--"]
          args:
            - "time python check.py --threads=3"
          volumeMounts:
            - name: cephfs-store
              mountPath: /mnt
      nodeSelector:
        node: {{ worker }}
      restartPolicy: Never
      volumes:
        - name: cephfs-store
          persistentVolumeClaim:
            claimName: subgrp1-subvol1-pvc
            readOnly: false
  backoffLimit: 0
{%- endfor %}

pip install jinja2  
alias render_template='python -c "from jinja2 import Template; import sys; print(Template(sys.stdin.read()).render());"' 
cat hdd-integrity.yaml.j2 | render_template | kubectl apply -f -

Schedule a job with replicated containers on different hosts

3 Answers3