GKE resource request can't be lowered past 100 mCPU

Question

I've recently set up a Kubernetes cluster using GKE Version 1.19.9-gke.1900 for the purpose of spinning up groups of microservices in different namespaces. Each service's pod contains a service image and an Istio proxy image. The initial CPU resource request for each pod is 100 mCPU, but this is greater than most of services in a given namespace will ever use. This has caused the cluster to run out of available CPU for services to launch, even while I'm only utilizing around 25% of the total CPU available at any given time.

Vertical Auto-Scaling is disabled and I've tried setting resource limits of 25 mCPU using the Deployment, ResourceQuota and LimitRange manifests.

# First tried this
apiVersion: apps/v1
kind: Deployment
spec:
  ...
  template:
    ...
    spec:
      containers:
        - name: service-name
          image: service-name:dev
          resources:
            requests:
              cpu: 25m
              memory: 128Mi
            limits:
              cpu: 2000m
              memory: 512Mi
      ...
---
# Then this
apiVersion: v1
kind: LimitRange
metadata:
  name: limit-range-limiter
spec:
  limits:
    - default:
        cpu: 100m
        memory: 512Mi
      defaultRequest:
        cpu: 25m
        memory: 128Mi
      type: Container
---
# Then this finally
apiVersion: v1
kind: ResourceQuota
metadata:
  name: resource-limiter
spec:
  hard:
    requests.cpu: 25m
    limits.cpu: 2000m
    requests.memory: 128Mi
    limits.memory: 512Mi

With the Deployment and LimitRange, instead of requesting just 25 mCPU for the service's pod, the pod will request 125 mCPU. And with the ResourceQuota, I receive an error that I must set the the cpu & memory requests and limits, despite it being in the manifest. This causes a replica failure and prevents my services from being launched.

I've also noticed each namespace has a ResourceQuota installed by default from GKE called gke-resource-quotas that can't be edited. But if I apply a ResourceQuota manifest directly after the namespace has been created but before any service get deployed, the gke-resource-quotas file doesn't get added.

Is it possible for pods to request anything lower than 100 mCPU on GKE?

EDIT: We managed to figure out what was going on. Istio was requesting 100 mCPU by default for the Pod. The actual config value was being set in the istio-sidecar-injector ConfigMap. Once we changed that value, the Pods were able to start with a lower value.

[here](https://cloud.google.com/kubernetes-engine/docs/concepts/pod), 100m is mentioned as the default value but nothing about a minimal value (even if 100m is mentioned as often too small for many apps). Do you really need to have only 2.5% of a CPU in your app? — guillaume blaquiere, Jul 16 '21 at 20:08
Sorry I misread your comment. The purpose here is to spin up the app in a minimum resourced state for development. More often than not, several services will not be utilized, thus they will not require much operational CPU. But they are there in case they are accessed inside the namespace. It's entirely possible for namespaces to use less than 2.5% CPU in their lifetime. — Alex Montolio, Jul 16 '21 at 23:12

score 0 · Answer 1 · answered Jul 20 '21 at 10:05

Would like to extend what guillaume blaquiere said,It’s possible for pods to request anything lower than 100 mCPU . For example see, Resource requests and limits

1- The default request for CPU is 100m. This is too small for many applications, and is probably much smaller than the amount of CPU available on the node.

2- There is no default request for memory. A Pod with no default memory request could be scheduled onto a node without enough memory to run the Pod's workloads.

3- Setting too small a value for CPU or memory requests could cause too many Pods or a suboptimal combination of Pods to be scheduled onto a given node and reduce performance.

4-Setting too large a value for CPU or memory requests could cause the Pod to be unschedulable and increase the cost of the cluster's resources.

For additional information refer.

So I understand that you in theory can create a pod with a Pod manifest file. But what im getting at is when using the Deployment manifests or any of the limiters, GKE does not start pods with the given cpu starting request. This seems to go against both the basic Kubernetes and GKE documentation. — Alex Montolio, Jul 21 '21 at 16:47

GKE resource request can't be lowered past 100 mCPU

1 Answers1