Getting KubeControllerManager, KubeProxy, KubeScheduler down alert in Kube Prometheus Stack installed in GKE

Question

I just installed the latest kube prometheus stack (kube-prometheus-stack-37.2.0) with default setting in my GKE cluster.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring

I started getting three alerts (Getting KubeControllerManager, KubeProxy, KubeScheduler down). By doing some research I found that I need to change the kube proxy's metricsBindAddress to 0.0.0.0:10249 in ConfigMap. But I can't find any kube-proxy or kube-proxy-config in kube-system namespace. Not sure what to do to fix this issue.

score 1 · Answer 1 · answered Jul 31 '22 at 08:46

SOLUTION: You just need to specify the right ports and additional options for both components. And everything will be fine and works!

Add the next directives to your values files:

## Component scraping the kube controller manager
##
kubeControllerManager:
  service:
    enabled: true
    ports:
      http: 10257
    targetPorts:
      http: 10257
  serviceMonitor:
    https: true
    insecureSkipVerify: "true"

## Component scraping kube proxy
##
kubeProxy:
  enabled: false
  service:
    enabled: true
    port: 10249
    targetPort: 10249

## Component scraping kube scheduler
##
kubeScheduler:
  service:
    enabled: true
    ports:
      http: 10259
    targetPorts:
      http: 10259
  serviceMonitor:
    https: true
    insecureSkipVerify: "true"

for kubeProxy, I have the same setup with enabled: false and same service configuration. I think it only disables the KubeProxyConfiguration, as soon as I set enabled: true, that is when the kubeproxy error start to pile up in my alerts. — Sameer Malhotra, Aug 01 '22 at 20:12

score 0 · Accepted Answer · answered Jul 19 '22 at 20:02

I couldn't find any answer on how to fix this issue of prometheus not being able to connect to KubeControllerManager, KubeProxy and KubeScheduler in GKE. So had to disable it for now in prometheus. Here is how I did it in case this helps someone else.

Create a yaml file (kube-prometheus-stack-overrides.yaml). Add the below content to this file:

## Component scraping the kube controller manager
##
kubeControllerManager:
  enabled: false
## Component scraping kube proxy
##
kubeProxy:
  enabled: false
  service:
    enabled: true
    port: 10249
    targetPort: 10249
## Component scraping kube scheduler
##
kubeScheduler:
  enabled: false

So basically, this file tells prometheus not to worry about these components (this is probably not a good idea but till we find a better solution, this will do!).

Execute the below command

helm upgrade -f kube-prometheus-stack-overrides.yaml [release name] prometheus-community/kube-prometheus-stack -n [namespace where stack is installed]

Getting KubeControllerManager, KubeProxy, KubeScheduler down alert in Kube Prometheus Stack installed in GKE

2 Answers2