5

i need a solution for that:

i have 2 node pools in gcloud kubernetes, first is preemptible and autoscaling, second is only autoscaling.

Jobs should be started on the first one ( with preemptible VMs ), but when no resources on the first pool are available Jobs should be started on the second one.

How can i realize that, maybe with Taints and Tolerations?

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
micha
  • 161
  • 1
  • 2
  • 14
  • Im also facing same problem. At the moment, `Estafette GKE Node Pool Shifter` could be temporary solution? https://cloud.google.com/blog/products/containers-kubernetes/cutting-costs-with-google-kubernetes-engine-using-the-cluster-autoscaler-and-preemptible-vms – tsu1980 Dec 27 '18 at 02:32
  • @tsu1980 `Estafette GKE Node Pool Shifter` is certainly interesting but currently lacks support for Regional clusters (at least on GKE). It only supports Zonal clusters and incorrectly requests too many new nodes. For example, for a 3-zone cluster 3 times as many nodes will be created than desired. This currently makes it unusable for our needs. – Martijn Heemels Jan 28 '20 at 08:45

1 Answers1

6

I don't think you can get exactly what you want with the Cluster Autoscaler but I'll hopefully give you a couple options and pointers to further explore.

  • The Cluster Autoscaler has the notion of Expanders which can help determine which node group to scale up when a scaling event happens. The price expander seems to be close to what you want, but based on the description of it, it doesn't look like it has support for preemptible VMs yet. You could explore that further, and possibly submit a feature request to add support for preemptible node pools.

  • When choosing a mixture of preemptible and non-preemptible nodes, whenever there is a stock out on GCP and preemptible nodes are not available, it's very likely that non-preemptible nodes will also not be available. In that case you may find yourself with a small number of non-preemptible nodes in the cluster and without the ability to create new ones.

    It may be a better idea to have a fixed minimum size of non-preemptible nodes, and auto-scale a preemptible node pool on top of that using the Cluster Autoscaler.

Ian Lewis
  • 917
  • 1
  • 7
  • 17