Prevent cluster autoscaling from killing pod

Question

I have a AWS EKS K8s setup with autoscaling group that uses spot instances.

Once a day there is a scheduled K8s cronjob which scales down a bunch of deployments, runs a script in a pod for 15min , then scales the deployments back up.

Issue is after deployments are scaled down, the script starts running and cluster autoscaling triggers to remove some nodes around 10min later which kills the script pod and moves it to different node.

This can't happen, how do I configure the cronjob script pod so that it never gets killed by autoscaler?

Thanks!

score 2 · Answer 1 · answered Oct 06 '21 at 18:57

This can't happen, how do I configure the cronjob script pod so that it never gets killed by autoscaler?

You can use the Node or POD affinity for scheduling the POD on specific Node so cronjob will not runt he PODs on Spot instances only you stateless workload will run on spot instance about which you don't care much.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: test
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - web-store
                topologyKey: "kubernetes.io/hostname"
          containers:
            - name: hello
              image: bash
              command: ["echo",  "Hello world"]
          restartPolicy: OnFailure

You can use the Node selector also to schedule the POD on specific type of Node group which is not scaling up & down.

Node affinity example

affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/e2e-az-name
            operator: In
            values:
            - e2e-az1
            - e2e-az2

Using the affinity you can plan on which node to schedule your POds.

Read more at : https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/

Thanks for the idea. Unfortunately for our setup there are only spot instances for now. The pod I don't want killed runs just for 15minutes, I found a solution to add this flag to the pod definition which should prevent autoscaler from killing it, testing it now. `"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"` — robliv, Oct 07 '21 at 14:52
Yes it will work there also option for PDB with it has few limitations better to this annotations only — Harsh Manvar, Oct 07 '21 at 16:10

Prevent cluster autoscaling from killing pod

1 Answers1