2

We have setup a kubernetes cluster on GCP with Istio (installed using Helm and Grafana is enabled) and it has Workload Identity enabled.

We are able to see Istio related dashboards in Grafana but when we create any new dashboard, folder, notification channel, etc. it doesn't remain persistent. When we restart the pod, all over custom creations get wiped out.

Can anyone suggest how to make them persistent? Do we need to modify grafana.persist option (if yes, from where can we do) or do we need to use PVC or something?

Thank you.

Hemal
  • 33
  • 1
  • 3
  • Check this: https://stackoverflow.com/a/57382663/2898919 – FL3SH Apr 12 '20 at 22:11
  • In order to persist your dashboards in Kubernetes, you should create _ConfigMap_ for each _dashboard.json_, see: https://rancher.com/docs/rancher/v2.x/en/monitoring-alerting/v2.5/persist-grafana/ – wissem mahjoub Apr 21 '21 at 09:53

1 Answers1

1

you can persist storage by mounting point at /var/lib/grafana.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - image: grafana/grafana:5.4.3
        name: grafana
        ports:
        - containerPort: 3000
          name: http

        volumeMounts:
          - name: grafana-storage
            mountPath: /var/lib/grafana
      volumes:
        - name: grafana-storage
          persistentVolumeClaim:
            claimName: grafana-storage
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        fsGroup: 472

or else if you have any existingClaim you can use it.

grafana:
  persistence:
    enabled: true
    storageClassName: prometheus
    existingClaim: prometheus-prometheus-operator-prometheus-db-prometheus-prometheus-operator-prometheus-0
Harsh Manvar
  • 27,020
  • 6
  • 48
  • 102
  • Thank you for the quick response. We have already tried first option i.e. mount PVC but with no luck. Grafana has been set up while installing Istio through Helm and thus we have everything inside /var/lib/grafana/dashboards/istio folder and it is persistent but if we try to create any new folder from Grafana UI, it gets wiped out. Also can you tell me from where can we make changes you mentioned in second option? – Hemal Apr 08 '20 at 11:47
  • 1
    just for ref : https://grafana.com/docs/grafana/latest/installation/configure-docker/ – Harsh Manvar Apr 08 '20 at 12:00
  • first create your pv or pvc for grafana, such as name is grafana-rwo, than modify prometheus-operator file if you have as second option. – Harsh Manvar Apr 08 '20 at 12:02
  • We have already tried these options. When Grafana is enabled while setting up Istio, it doesn't allow to persist any other dashboard except Istio. Is there anything like we need to make changes in Grafana's configuration (grafana.ini or default.ini)? – Hemal Apr 10 '20 at 11:26