How to configure kafka chart to expose both: kafka-metrics and kafka-jmx-metrics to prometheus?

Question

We use kafka chart (version 17) as a dependency of our chart. And for I try to configure monitoring of that kafka.

values.yaml:

kafka:
  metrics:
    kafka:
      enabled: true
    jmx:
      enabled: true
    serviceMonitor:
      enabled: true
      labels:
        my.custom.label/service-monitor: "1.0"
   ...

Then I deploy my application and go to prometheus targets and I see following:

I am concerned about the failed request.

What I see ?

There are 2 services to monitor:

jmx-exporter
kafka-exporter

and both of those services are requested by 2 paths:

/
/metrics

But kafka-exporter doesn't respond on /. Is there way to configure kafka chart to work without failures if there are 2 exporters are enabled ?

P.S.

Services definition:

spec:
  endpoints:
    - path: /
      port: http-metrics
  namespaceSelector:
    matchNames:
      - my-ns
  selector:
    matchLabels:
      app.kubernetes.io/component: metrics
      app.kubernetes.io/instance: dev-my-app
      app.kubernetes.io/name: my-app-kafka

spec:
  endpoints:
    - path: /metrics
      port: http-metrics
  namespaceSelector:
    matchNames:
      - my-ns
  selector:
    matchLabels:
      app.kubernetes.io/component: metrics
      app.kubernetes.io/instance: dev-my-app
      app.kubernetes.io/name: my-app-kafka

Why is `kafka-exporter` underneath `kafka-jmx-metrics` target? I thought `kafka-exporter` had lag and other metrics, **not** JMX — OneCricketeer, Jul 22 '22 at 16:16
@OneCricketeer sorry, didn't get you. Could you please rephrase ? — gstackoverflow, Jul 22 '22 at 16:19
In Prometheus UI, it is grouping your targets under a label. The two targets at the bottom if your image are correct, and are `kafka-exporter`. Only one of the ones at the top is the jmx-exporter (see `container` label) — OneCricketeer, Jul 22 '22 at 16:20
Still didn't get you. `Target 1:` a) kafka-exporter b) jmx-exporter `Target 2:` a) jmx-exporter b) kafka-exporter — gstackoverflow, Jul 22 '22 at 16:25
`ServiceMonitor/...kafka-jmx-metrics`. Look at the table below. Look at `Labels` column. Look at blue `service` or `container` labels. Each of those underneath that serviceMonitor should only say `jmx-exporter` or `kafka-jmx-metrics` — OneCricketeer, Jul 22 '22 at 16:29
Yes, agree with you. Because 2 service monitors has the same selector. I checked it. — gstackoverflow, Jul 22 '22 at 16:32

OneCricketeer · Answer 1 · 2022-07-22T17:46:47.967

1

kafka-exporter doesn't respond on /.

That's because web.telemetry-path defaults to /metrics

https://github.com/danielqsj/kafka_exporter/blob/master/kafka_exporter.go#L700

Your ServiceMonitors are mixed up

the one that ends in kafka-jmx-metrics should have container=jmx-exporter labels
one that ends with kafka-metrics should have container=kafka-exporter

Currently, you've shown there is one of each container in the opposite ServiceMonitors. I suspect this is because your label selectors are the exact same, but for different paths.

If you only have one Kafka broker, you should only have 2 targets. One JMX Exporter (for each broker) and one Kafka exporter.
So, two brokers would have 3 targets: 2 JMX Exporters and one Kafka exporter (since it is for the whole cluster).

edited Jul 22 '22 at 17:46

answered Jul 22 '22 at 16:21

OneCricketeer

179,855
19
132
245

I've read about this variable. Can I override it somehow to avoid error ? – gstackoverflow Jul 22 '22 at 16:21
There should be nothing to override for the exporter. The ServiceMonitor definition is what has the path. – OneCricketeer Jul 22 '22 at 16:26
As you can see I use service monitor from kafka chart. Should I write it on my own ? – gstackoverflow Jul 22 '22 at 16:30
Maybe. Or you should open a Github issue/Pull-Request against Bitnami repo – OneCricketeer Jul 22 '22 at 16:31
Why not just change `path: /` to have `/metrics`? I don't understand what is running that has metrics on just `/` – OneCricketeer Jul 22 '22 at 17:13
could you please provide details how to achieve it ? I am novice in helm so I can miss smth obvious. – gstackoverflow Jul 22 '22 at 17:19
It's not really a Helm change. You should be able to use `kubectl edit` directly on the ServiceMonitor resource, for example – OneCricketeer Jul 22 '22 at 17:34
But if you wanted to edit the Helm chart, its right here https://github.com/bitnami/charts/blob/master/bitnami/kafka/templates/servicemonitor-jmx-metrics.yaml#L34 – OneCricketeer Jul 22 '22 at 17:39
I think kubectl edit is a bad option. – gstackoverflow Jul 22 '22 at 17:51
As long as you're not re-running `helm install`, then it is a fine option – OneCricketeer Jul 22 '22 at 18:06
Could you please provide more details? – gstackoverflow Jul 22 '22 at 18:14
The resources will never change until you run `helm install` again. In turn, `helm install` runs multiple `kubectl apply` commands. But you are only changing one resource, so you only need to run `kubectl apply` one time, which is what `kubectl edit` will do when you close the editor – OneCricketeer Jul 22 '22 at 18:16
We use gitlab piprline to install chart. From my point of view kubectl edit is a manual step – gstackoverflow Jul 22 '22 at 18:19
Alright, well, I cannot tell you how to manage your k8s resources. All I know is that something with your `path` or `matchLabels` are wrong. The [JMX Exporter](https://github.com/prometheus/jmx_exporter) documentation also says the path is `/metrics`, so based on the link above to the Bitnami chart, the hard-coded path is simply incorrect and you need to edit the resource on your own... – OneCricketeer Jul 22 '22 at 18:22
1

I've implemented my own service monitors and error is disappeared but anyway I see duplicated requests because both service monitors has the same selectors. I've checked services for kafka-exporter and jmx-exporter. They have exactly the same labels so I have no Idea how to distinguish them. looks like services are configured here https://github.com/bitnami/charts/blob/master/bitnami/kafka/templates/kafka-metrics-svc.yaml How can I can have different lables for services ? – gstackoverflow Jul 26 '22 at 13:36
Looks like only annotations can be different. As suggested, raise an issue in that repo and/or submit a PR. Otherwise, fork the chart on your own and make your own changes to add new Values section for service-specific metric labels – OneCricketeer Jul 26 '22 at 14:36
is it possible to override values on my side somehow ? – gstackoverflow Jul 26 '22 at 14:40
Only if you manually edit those files you've linked to before running `helm install`. You cannot change the values alone since there's no template to load anything differently – OneCricketeer Jul 26 '22 at 14:50
I meant... I add k8s template to my project and i expect it overrides kafka template somehow. Not sure if it is possible – gstackoverflow Jul 26 '22 at 14:55
You'd still need to open those `svc` yaml files in some text editor – OneCricketeer Jul 26 '22 at 14:56
Maybe `kustomize` is another option you can look into – OneCricketeer Jul 26 '22 at 14:57
I would appreciate if you could provide a bit more details about each option. – gstackoverflow Jul 26 '22 at 15:08
Both have the same result. Kustomize is explicitly meant to patch resources with static values. Otherwise, you need to manually edit the Helm template – OneCricketeer Jul 26 '22 at 15:30
I am asking about implementation details – gstackoverflow Jul 26 '22 at 15:33
Eventually I fixed it in that way: https://stackoverflow.com/a/73151165/2674303 – gstackoverflow Jul 28 '22 at 10:19

score 1 · Accepted Answer · edited Jul 28 '22 at 10:25

1

Eventually I finished with following solution:

I use jmx exporter from kafka chart:

values.yaml

kafka:
  metrics:
    kafka:
      enabled: false 
    jmx:
      enabled: true
    serviceMonitor:
      enabled:true
      labels:
       my.super.company/service-monitor: "1.0"

  enabled: true

and registered kafka exporter as separated dependency:

chart.yaml:

  - name: prometheus-kafka-exporter
    version: 1.6.0
    repository: https://prometheus-community.github.io/helm-charts

and configured it manually:

values.yaml

prometheus-kafka-exporter:
  kafkaServer:
    - my-kafka-server:9092
  prometheus:
    serviceMonitor:
      enabled: true
      namespace: ""
      additionalLabels:
        my.super.company/service-monitor: "1.0"

edited Jul 28 '22 at 10:25

OneCricketeer

179,855
19
132
245

answered Jul 28 '22 at 10:18

gstackoverflow

36,709
117
359
710

Where is `.kafkaServer` value list used? – OneCricketeer Jul 28 '22 at 10:26
it is used inside prometheus-kafka-exporter helm chart I suppose – gstackoverflow Jul 28 '22 at 10:32
1

it is defined at: kafka.kafkaSource.brokers – gstackoverflow Jul 28 '22 at 10:33
Oh, my bad. I was thinking that was just the Prometheus operator itself, and completely read over `kafka-exporter` – OneCricketeer Jul 28 '22 at 12:59

How to configure kafka chart to expose both: kafka-metrics and kafka-jmx-metrics to prometheus?

P.S.

2 Answers2