0

I am using prometheus federation to scrape metrics from multiple k8s cluster. It works ok and I would like to create some dashboard on grafana which I'd like to filter dashboards by tenant(cluster).. I am trying to use variables but the things that I do not understand, even if ı did not specify something special for kube_pod_container_status_restars_total,it contains the label that I specied below static_configs but kube_node_spec_unschedulable is not.

So where this differences come from and what should I do ? Meanwhile what is the best practice way to setup a dashboard which can provide dashboard filter by multiple cluster name? should ı use relabel?

kube_pod_container_status_restarts_total{app="kube-state-metrics",container="backup",....,tenant="022"}

kube_node_spec_unschedulable{app="kube-state-metrics",....kubernetes_pod_name="kube-state-metrics-7d54b595f-r6m9k",node="022-kube-master01",pod_template_hash="7d54b595f"

Prometheus Server

prometheus.yml:
  rule_files:
    - /etc/config/rules
    - /etc/config/alerts

  scrape_configs:
    - job_name: prometheus
      static_configs:
        - targets:
          - localhost:9090

Central Cluster

  scrape_configs:
    - job_name: federation_012
      scrape_interval: 5m
      scrape_timeout: 1m

      honor_labels: true
      honor_timestamps: true
      metrics_path: /prometheus/federate

      params:
        'match[]':
          - '{job!=""}'
      scheme: https

      static_configs:
        - targets:
          - host
          labels:
            tenant: 012

      tls_config:
        insecure_skip_verify: true

    - job_name: federation_022
      scrape_interval: 5m
      scrape_timeout: 1m

      honor_labels: true
      honor_timestamps: true
      metrics_path: /prometheus/federate

      params:
        'match[]':
          - '{job!=""}'
      scheme: https

      static_configs:
        - targets:
          - host
          labels:
            tenant: 022

      tls_config:
        insecure_skip_verify: true
semural
  • 3,583
  • 8
  • 37
  • 67
  • probably you need to try `honor_labels=true`. In case it does not affect the labels try `relabel_configs`. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config – dmkvl Apr 24 '20 at 22:21
  • @dmkvl I've already honor_labels_true.. Where and how should ı add this relabel_configs? – semural Apr 25 '20 at 07:04
  • I don't see it in your scrape_configs, could you share your full/updated scrape_configs? – dmkvl Apr 25 '20 at 07:06
  • @dmkvl, ı added whole in answer.. should ı need to add relabel_configs in each prometheus server or for the federation one ? And how should add to differentiate to tenant name? – semural Apr 25 '20 at 10:42
  • I posted an answer with the possible `federate` and `source/tenant` scrape jobs configs. I suppose this should help. – dmkvl Apr 25 '20 at 12:04

1 Answers1

1

Central Prometheus server 

  scrape_configs:
    - job_name: federate
      scrape_interval: 5m
      scrape_timeout: 1m

      honor_labels: true
      honor_timestamps: true
      metrics_path: /prometheus/federate

      params:
        'match[]':
          - '{job!=""}'
      scheme: https

      static_configs:
        - targets:
          - source_host_012
          - source_host_022

      tls_config:
        insecure_skip_verify: true

Source Prometheus (tenant 012)

prometheus.yml:
  rule_files:
    - /etc/config/rules
    - /etc/config/alerts

  scrape_configs:
    - job_name: tenant_012
      static_configs:
        - targets:
          - localhost:9090
          labels:
            tenant: 012

Source Prometheus (tenant 022)

prometheus.yml:
  rule_files:
    - /etc/config/rules
    - /etc/config/alerts

  scrape_configs:
    - job_name: tenant_022
      static_configs:
        - targets:
          - localhost:9090
          labels:
            tenant: 022

If you still don't get needed labels, try to add relabel_configs to you federate job and try to differentiate metrics by a source job name:

relabel_configs:
  - source_labels: [job]
    target_label: tenant

or extract distinctive information from the __address__ (or from any other __ prefixed) label for example.

relabel_configs:
  - source_labels: [__address__]
    target_label: tenant_host

PS: keep in mind that labels starting with __ will be removed from the label set after target relabeling is completed.

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

dmkvl
  • 732
  • 5
  • 13
  • ı will try and let you know the results but per my understanding ı need to add relabel_configs for federated job right not for source prometheus? – semural Apr 25 '20 at 12:08
  • first try to configure source prometheus servers as in the example above, it seems that it will work without `relabel_configs` – dmkvl Apr 25 '20 at 12:12
  • ı follow your instructions but it wont work.. Now kube_pod_container_status_restarts_total also do not have the tenant label – semural Apr 26 '20 at 07:45
  • So give an example of your `kube_pod_container_status_restarts_total` metric with labels – dmkvl Apr 26 '20 at 11:29
  • 'kube_pod_container_status_restarts_total{app="kube-state-metrics",container="backup",job="kubernetes-pods",kubernetes_io_hostname="022-kube-master01",kubernetes_namespace="kube-system",kubernetes_pod_name="kube-state-metrics-7d54b595f-r6m9k",namespace="database",node="022-kube-master01",pod="postgresql-postgresql-helm-backup-1587686400-7mczw",pod_template_hash="7d54b595f",ready="true"}' – semural Apr 26 '20 at 15:09
  • @semural, you could extract the tenant id/name from the labels `kubernetes_io_hostname` or `node` – dmkvl Apr 28 '20 at 15:37