Issue with Adding the additional scrapeconfigs in kube-prometheus-stack

Question

I've Installed kube-prometheus-stack through helm chart. Need to add additional scrape configs for prometheus. Created a configmap to scrape the metrics from grok-exporter

 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: prometheus
 data:
   prometheus.yml: |-
     global:
       scrape_interval: 15s
     scrape_configs:
     - job_name: 'grok'
       static_configs:
     - targets: ['grok-exporter:9144']

Applied this configmap. Then created the secret from this configmap using the below command

       ""kubectl create secret generic grok-prometheus --from-file=grok-prometheus.yaml"

Secret is created. Then Added additionalScrapeConfigSecrets in the values.yaml of kube-prometheus-stack.

 additionalScrapeConfigsSecret:
   enabled: true
   name: grok-prometheus
   key: grok-prometheus.yaml

After this upgraded the helmchart

when I check the "kubectl get prometheus -o yaml" able to see the additionalScrapeConfigs are added.

  spec:
additionalScrapeConfigs:
  key: grok-prometheus.yaml
  name: grok-prometheus

But I got the below error in prometheus output.

- lastTransitionTime: "2022-07-30T16:45:41Z"
  message: |-
    creating config failed: generating config failed: generate additional scrape configs: unmarshalling additional scrape configs failed: yaml: unmarshal errors:
      line 1: cannot unmarshal !!map into []yaml.MapSlice
  reason: ReconciliationFailed
  status: "False"
  type: Reconciled

Can anyone help me with this. Thanks in advance.

score 0 · Answer 1 · answered Jul 31 '22 at 19:34

0

Seems like indentation issue in the ConfigMap. targets should be one level deeper than static_configs. Try this YAML.

 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: prometheus
 data:
   prometheus.yml: |-
     global:
       scrape_interval: 15s
     scrape_configs:
     - job_name: 'grok'
       static_configs:
       - targets: ['grok-exporter:9144']

answered Jul 31 '22 at 19:34

Emruz Hossain

4,764
18
26

After correcting the Indentation also getting the same error – Hema Sri Aug 03 '22 at 04:52

Adam Malik · Answer 2 · 2022-08-31T07:10:51.220

The secret should not be a ConfigMap, but a instead the scrape config entry, like documented here: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md

- job_name: 'grok'
   static_configs:
 - targets: ['grok-exporter:9144']

Here is a better documentation: https://github.com/prometheus-community/helm-charts/blob/8b45bdbdabd9b54766c4beb3c562b766b268a034/charts/kube-prometheus-stack/values.yaml#L2691

According to this, you could add scrape configs without packaging into a secret like this:

additionalScrapeConfigs: |
  - job_name: "prometheus"
  static_configs:
  - targets: ["localhost:9090"]

Issue with Adding the additional scrapeconfigs in kube-prometheus-stack

2 Answers2