1

I had installed kube-prometheus-stack from the helm chart repo prometheus-community

(k8s: minikube) $ kubectl get deploy,statefulset -n monitoring
NAME                                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-prometheus-stack-grafana              1/1     1            1           20d
deployment.apps/kube-prometheus-stack-kube-state-metrics   1/1     1            1           20d
deployment.apps/kube-prometheus-stack-operator             1/1     1            1           20d

NAME                                                               READY   AGE
statefulset.apps/alertmanager-kube-prometheus-stack-alertmanager   1/1     20d
statefulset.apps/prometheus-kube-prometheus-stack-prometheus       1/1     20d

As you can see, by default, grafana installed as Deployment, but I want to change the kind to Statefulset by changing it in its helm chart, instead of direct kubectl edit on the cluster.

Following is the directory structure inside the kube-prometheus-stack repo:

kube-prometheus-stack vjwilson(k8s: minikube) $ ls
Chart.lock  charts  Chart.yaml  CONTRIBUTING.md  crds  README.md  templates  values.yaml

kube-prometheus-stack (k8s: minikube) $ tree -d
.
├── charts
│   ├── grafana
│   │   ├── ci
│   │   ├── dashboards
│   │   └── templates
│   │       └── tests
│   ├── kube-state-metrics
│   │   └── templates
│   └── prometheus-node-exporter
│       ├── ci
│       └── templates
├── crds
└── templates
    ├── alertmanager
    ├── exporters
    │   ├── core-dns
    │   ├── kube-api-server
    │   ├── kube-controller-manager
    │   ├── kube-dns
    │   ├── kube-etcd
    │   ├── kubelet
    │   ├── kube-proxy
    │   └── kube-scheduler
    ├── grafana
    │   └── dashboards-1.14
    ├── prometheus
    │   └── rules-1.14
    └── prometheus-operator
        └── admission-webhooks
            └── job-patch

30 directories

I am confused and stuck where exactly on this helm to change and tell grafana to install as Statefulset instead of default Deployment. Would be great if someone can help with it.

vjwilson
  • 754
  • 2
  • 14
  • 30
  • Which version of Kubernetes did you use and how did you set up the cluster? Did you use bare metal installation or some cloud provider? – kkopczak Feb 03 '22 at 13:38
  • It is a new cluster we setup for managing infra. version is 1.22.4. Cluster is created using `kops`, the underlying master nodes and worker nodes are EC2 instances of AWS. – vjwilson Feb 03 '22 at 16:04

2 Answers2

2

Here's how I found the answer. In a helm chart, if there is a folder named charts, that means that the chart is declaring chart dependencies. Looking at the Chart.yaml, we see the grafana dependency:

dependencies:
- name: grafana
  version: "6.21.*"
  repository: https://grafana.github.io/helm-charts
  condition: grafana.enabled

Going to this link, We can look at their statefulset.yaml. Looking here we find that Grafana creates a stateful set using this condition:

{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) (eq .Values.persistence.type "statefulset")}}

A dependent chart can still have its chart values overwritten if you have a section in your values.yaml that has a top level tag of the dependency name. So in this case, the dependency is named grafana, so we can override the values.yaml of the dependent chart using this configuration:

grafana:
  enabled: true
  persistence:
    enabled: true
    type: statefulset

(For other configuration options see this repo. All of the values.yaml from this chart can be overwritten as long as they are inside of the grafana: block.)

The dependent chart is the official chart from Grafana. However, if this doesn't work for you (maybe you aren't using persistent volume claims), your second option is to disable the grafana dependency from the chart you are using and deploy a custom version of the Grafana chart.

grafana:
  enabled: false

Once you disable grafana, you can then install grafana on its own and either alter the generated manifests using something like Kustomize or a simple sed replace, or fork the grafana helm chart and use your own custom grafana chart that is deployed as a statefulset.

R10t--
  • 803
  • 7
  • 16
  • Are you saying that I can tell the grafana values.yml to use the statefulset.yaml template instead of deployment.yaml. If so, how exactly to do that?. – vjwilson Feb 03 '22 at 18:41
  • Correct. In the `values.yaml` for the kube-prometheus-stack helm chart, just add the "grafana: enabled: true ..." code block that I have shown above that enables persistence. That should tell the dependent grafana chart that you want to deploy it as a statefulset instead of the default. – R10t-- Feb 03 '22 at 21:46
  • 1
    Great, that works really fine. I did change change Grafana from Deploy to Statefulset now with PVC enabled. – vjwilson Feb 05 '22 at 16:43
1

Enable persistence if you want to make it stateful. However I did not see an option to make Grafana a statefulset in the chart you mentioned.

Usually you will see persistence enable option if the corresponding Helm chart support it. For example: you can enable persistence in this grafana helm chart. You may generate template out of it and make use of it in your repo.

ffran09
  • 887
  • 7
  • 9
  • Thank you, I had already enabled persistence on Grafana deployment and the PVC and PV is `ReadWriteOnce`. But this might be an issue when we do scale-up in the future and when pods created on other worker nodes – vjwilson Feb 03 '22 at 04:08
  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/low-quality-posts/30974617) – Artem Feb 03 '22 at 06:28
  • 1
    @Artem I have made changes in my answer to better describe, however I am not sure if I can copy paste the content in a meaningfulway. – ffran09 Feb 03 '22 at 08:21
  • @JohnW I think it depends on where your cluster is and you might be ok if it is single replica grafana – ffran09 Feb 03 '22 at 08:30
  • Yep, I understand keeping a single Grafana deployment replica and PVC with it is ok, but in case we scale up in the future, the current setup would be a problem as new pods may try to mount on the same volume existing pods share. – vjwilson Feb 03 '22 at 09:11