16

I have a multi-container pod in my kubernetes deployment:

  • java
  • redis
  • nginx

For every of those containers, there's a container with Prometheus exporter as well.

The question is how can I expose those ports to Prometheus if annotations section supports only one port per pod?

annotations:
  prometheus.io/scrape: 'true'
  prometheus.io/port: 'xxxx'

but I need something like this:

annotations:
  prometheus.io/scrape: 'true'
  prometheus.io/port_1: 'xxxx'
  prometheus.io/port_2: 'yyyy'
  prometheus.io/port_3: 'zzzz'

Maybe there's some other method to scrape all metrics from my multi-container pods? Thanks in advance for any kind of help.

d.ansimov
  • 2,131
  • 2
  • 31
  • 54

2 Answers2

12

Here's an example job for Prometheus. Put it in your own config.

Next, add:

annotations:
   prometheus.io/scrape: 'true'

to your pod metadata.

And on every container, which provides /metrics to prom, create an appropriate port, named metrics.

That's it. Prometheus would scrape only those ports, and there would be no situation, like when your redis instance would get http requests on its 6379 port.

d.ansimov
  • 2,131
  • 2
  • 31
  • 54
  • To be clear: In the deployment yaml add `containerPort: "metrics"` and in the code itself set the prometheus server to start on the "metrics" port? And the Dockerfile also needs to have `EXPOSE metrics` – north.mister Feb 26 '18 at 23:14
  • Expose your port with prometheus metrics via your app dockerfile, add `- name: metrics containerPort: 1234` to a container at your deployment, `annotations: prometheus.io/scrape: 'true'` to this pod metadata. That's it. You don't need to *set the prometheus server to start on the "metrics" port*. – d.ansimov Feb 27 '18 at 08:58
  • apologies. I meant the internal prometheus metrics reporter. (The one that you start in the code to report metrics back to the master server). Thanks for the clarification – north.mister Feb 27 '18 at 22:36
  • 4
    An example yaml would be really helpful. – himanshu Jun 01 '21 at 09:26
  • @himanshu did you get an example ? – user9920500 Jan 12 '23 at 10:59
0

The annotations you propose should work. Create one scrape_config per port annotation, keeping only targets matching the corresponding annotation port name.

brian-brazil
  • 31,678
  • 6
  • 93
  • 86
  • Thanks for a suggestion, mate. It seems like if I set only `annotations: prometheus.io/scrape: 'true'`, Prometheus would scrape on all exposed ports. So, it's enough for me. – d.ansimov Mar 31 '17 at 11:35
  • Hi d.ansimov, Does it work for you ?? I mean the syntax you proposed ??? – user9920500 Jan 10 '23 at 15:13
  • @OlivierDulac that is of less priority now, what I am looking for is a way to fix the problem – user9920500 Jan 11 '23 at 13:06