As we can get information from doc, the purpose of auto generated secret is for recording release information. In k8s design, Individual secrets are limited to 1MiB in size. Based on the above information, the secret size is the hard limitation of k8s, and the actual release secret size should be positively correlated with the size of the helm chart.
In this use case, the main reason for the large helm chart is that you use grafana's dashboardProvider to automatically deploy the ready-made dashboard JSON file. The provider will load all JSON file to kube-prometheus-stack for creating dashboard configmaps. And then in one day when you add new dashboard and it makes release secret finally hit the limitation you will get the error.
If you don't want to change k8s storage backend type, there is alternative way to work around with. The main idea is to separat tasks of creating dashboard configmap from grafana dashboardProvider and create dashboard configmap by our own.
First, we can abandon this kind of declaration in kube-prometheus-stack
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- name: 'default'
orgId: 1
folder: 'default'
type: file
disableDeletion: true
editable: true
options:
path: /var/lib/grafana/dashboards/default
dashboards:
default:
{{- range $_, $file := ( exec "bash" (list "-c" "echo -n dashboards/default/*.json") | splitList " " ) }}
{{ trimSuffix (ext $file) (base $file) }}:
json: |
{{- readFile $file }}
{{- end }}
Then, we create another helm chart configmap
Helm chart template
{{- range $config, $data := .Values.configs }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-dashboard-{{ $config }}
labels:
grafana_dashboard: "1"
annotations:
grafana_folder: {{ $config }}
data:
{{ range $key, $val := $data }}
{{ $key }}.json: |
{{ mustToJson $val }}
{{ end }}
{{- end }}
Helm values, read dashboard json file and convert to golang string
configs:
default:
{{- range $_, $file := ( exec "bash" ( list "-c" (printf "echo -n dashboards/default/*.json")) | splitList " ") }}
{{ trimSuffix (ext $file) (base $file) }}:
{{ readFile $file }}
{{- end}}
At this time, when we deploy this separated dashboard helm chart, it should generate all configmaps which contain dashboard json value automatically.
Finally, the last step, we can go to setup Grafana sidecar configurations to make it scrape dashboard from configmaps.
grafana:
defaultDashboardsEnabled: false
sidecar:
dashboards:
enabled: true
label: grafana_dashboard
annotations:
grafana_folder: "Default"
folder: /tmp/dashboards
folderAnnotation: grafana_folder
provider:
foldersFromFilesStructure: true
After update kube-prometheus-stack and waiting for a while, or you can monitoring on Grafana sidecar pod logs. You will see the dashboard configmaps are loading to pod and ADD to dashboard.
