2

My question is an addition to Prometheus dns service discovery in docker swarm.

I define the prometheus scrape targets as follows:

- job_name: 'node-exporter'
  dns_sd_configs:
  - names:
    - 'tasks.nodeexporter'
    type: 'A'
    port: 9100

This works fine but results in prometheus using the IP of the docker container as instance label.

I tried to relabel the instance label as follows:

relabel_configs:
- source_labels: [__meta_dns_name]
  target_label: instance

But doing so results in all instances of node-exporter having the same label "tasks.nodeexporter".

Is it somehow possible to relabel the instance label to something like tasks.nodexporter_1, tasks.nodeexporter_2, ...?

Andreas Jägle
  • 11,632
  • 3
  • 31
  • 31
Yannic Bürgmann
  • 6,301
  • 5
  • 43
  • 77

1 Answers1

2

Service discovery for docker swarm setups isn’t supported well in Prometheus as there are many features missing on the swarm side.

The dns service discovery is one way to mitigate those missing features but in my opinion it isn’t a good solution and I recommend to not use it in production:

  • there is no way to provide additional information e.g. using SRV records
  • there is no information about how many instances should be running
  • as the the dns only lists the healthy tasks, the amount of scrape targets decreases when one task is no longer considered healthy which makes it harder to alert on misbehaving containers
  • when containers die and are restarted you will observe new instances as there is no information like the task slot available

Altogether these issues don’t allow this approach for being a reliable source for a monitoring system.

If you are really tied to using docker swarm, you should consider building a more sustainable solution by querying the docker api programmatically and using the file_sd service discovery mechanism of Prometheus. Please see this poc by containersolutions for reference: https://github.com/ContainerSolutions/prometheus-swarm-discovery

Andreas Jägle
  • 11,632
  • 3
  • 31
  • 31
  • Thank you for this hint. I will have a look at it and think about it. For now this sounds like the better approach and even solves my problem since the docker task name is already included in the labels. This answer will be accepted after I tried to implement it. – Yannic Bürgmann Jun 30 '18 at 14:27