0

I have a diverse set of artifacts that require processing and a non-homogenous computing environment, e.g. - some inputs are large and take a long time and lots of memory, others are small so take a short time and little memory, and the nodes do not have a single set number of cores and memory.

I would like a simple set of greedy rules regarding the adding of pods:

If there are cores remaining, add pods with the constraint among inputs to process that chooses the largest input that will fit in remaining memory.

So far, I have found it difficult to express this using the Horizontal Pod Autoscaler average and target CPU usage, especially due to the semantic meaning of average utilization referring to processes within pods. At this point, I think it is required to use custom metrics or some way to integrate the selection of arguments with kubernetes.

But I don't know what the best way forward with this is. If I were to program this to integrate with kubernetes, what API would I use? It would be great if I could register a callback with kubernetes, receive metrics information where different signals could be given to kubernetes for adding pods or removing them by simply returning a type or signal or calling another API.

RadekW
  • 439
  • 1
  • 11
Adam Miller
  • 1,756
  • 1
  • 25
  • 44
  • What kubernetes version is used? There are some differences in HPA in fresh versions – moonkotte Jan 05 '22 at 15:13
  • 1
    $ kubectl version Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"} – Adam Miller Jan 05 '22 at 19:48
  • Could you please clarify in more/exact details `difficult to express this using the Horizontal Pod Autoscaler average and target CPU usage`? What exactly you tried and why it doesn't work fine for you? Also if your question is about placing scheduling pods to some specific nodes, that's the question about `scheduler` and particularly [pods assigning to nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). – moonkotte Jan 10 '22 at 13:47
  • Basically, the HPA has average utilization but the selection of work to dispatch based on currently free metrics isn't a feature of kubernetes yet. I would have to poll the metrics API to get that with my broker/distributor. So far, I have found the kubectl get NodeMetrics command to return some absolutely incorrect information for my nodes. Honestly, I tried so many different things with the HPA and never could get it to scale up more than the minimum number of pods. – Adam Miller Jan 11 '22 at 15:29
  • Or for that matter to deploy to other nodes than what it was already on. – Adam Miller Jan 11 '22 at 15:30
  • But still, can you solve your question using [`nodeAffinity`](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) into your deployments? Idea is to separate deployments into heavy ones which require a lot of resources and schedule them on big nodes and smaller ones for smaller nodes. – moonkotte Jan 12 '22 at 11:43
  • No. I opted to instead use nodeName field, and write a loop and create deployments or other things per node. – Adam Miller Jan 19 '22 at 14:37

0 Answers0