0

Why should K8S Deployments/Pods set Memory Limit (which seems to be a recipe to find yourself in a crash loop) rather than each Deployment simply being responsible for setting its own Memory Request high enough that it won't be killed?

vmayer
  • 985
  • 2
  • 9
  • 18
  • Request is different from Limit. Request asks the K8s controller for some memory. Limit prevents a deployment from going over a certain amount. You should scope to Request when you know how much memory is expected. – ryanwebjackson Jul 20 '23 at 22:09
  • [Request vs limit cpu in kubernetes/openshift](https://stackoverflow.com/questions/54819381/request-vs-limit-cpu-in-kubernetes-openshift) has quite a bit of explanation; is what you need there? The general motivation for setting limits on individual Pods is to let something leaking memory be terminated before it starts impacting other processes; you'd prefer to not involve the kernel OOM killer. – David Maze Jul 21 '23 at 00:40
  • So, if Pods A, B, and C are all running on the node, and A and B set Request Mem high enough, then they are guaranteed not to be killed. If C has a memory leak, it will be the one to get killed anyway. Right? Am I missing something there? – vmayer Jul 21 '23 at 04:46
  • Mainly, I'm wondering about applications that are not able to scale horizontally and can only scale vertically and have memory that grows with the size of their inputs (as opposed to a memory leak). The upper limit of memory is unknown. (And all applications have meanwhile been asked to set Limits.) – vmayer Jul 21 '23 at 04:50

1 Answers1

1

By setting requests but not limits you get the Quality of Service (QOS) Burstable. Without limits set your pod is able to consume all "available" cpu and memory on a node. "Available" resources are those that are not allocated to other pods via requests. This does NOT include resource limits. What you should consider is whether the process in the container will try to use more memory than your request. If for instance you have a java application then you will want to set the max heap size (Xmx) to less than the limit. But not everything that people run in containers has this type of memory control.

If you are sure you container only needs x amount of memory then what you want is the Guaranteed QOS. i.e set limit and request to the same value. It sounds like your container is trying to use more than your memory request. If it is, you should consider something similar to the java max heap size configuration.

Setting a memory limit is a safeguard (if you need a safe guard i.e you don't want the pod evicted unpredictably). A good use case for burstable QOS is a pod that needs a bit of extra memory "sometimes" e.g e-commerce site for coffee that always gets busy between 7am-9am but is quite for the rest of day.

Isaac Carrington
  • 56
  • 5
  • Thanks for the detailed response. You mentioned "This does NOT include resource limits." It sounds like you're saying that if Pod A indicates request=x and limit=y, then Pod B can use everything except x, so it can use everything between y and x. This seems okay since technically only x is guaranteed. Sorry if I'm missing something. – vmayer Jul 21 '23 at 17:59
  • The concern is that our application does not have a known memory limit, as the size of its memory increases with the amount of work it needs to do. And it can't scale horizontally. (I recognize this means that at some point it could exceed the size of the node itself, in which we'd have to take other measures.) But this is the background on why we prefer not to have an artificial Limit. – vmayer Jul 21 '23 at 18:05
  • 1
    "This does NOT include resource limits." is in regards to the scheduler determining if there is enough resource on a node to schedule a pod. It basically tally's up all the requests to determine this, not the limits. Coming back to your opening question "Why should K8S Deployments/Pods set Memory Limit". You do it, or don't, so that you get a quality of service of guaranteed, burstable, or best effort. An application inside the container always has a responsibility of staying within said requests or limits if it doesn't want to be killed or throttled (cpu only). Does this help? – Isaac Carrington Jul 22 '23 at 03:33
  • Thanks. I guess it sounds like Limits are best for Pods that are concerned about a memory leak, whereby a restart would probably be the best thing for it. (And I can see how being in the Guaranteed class can allow for preferential treatment when resources are low.) For our application, in which memory is correlated to the amount of work we're doing, and if we did get an OOMKilled we would simply restart and immediately re-populate all of that same memory and get OOMKilled again, we probably should either not set a limit or consider a strategy of offloading to disk. – vmayer Jul 24 '23 at 15:59