Docker - Prometheus container dies immediately

Question

I have cadvisor running with port mapping 4000:8080 and I have to link it with a container with prometheus.

My prometheus.yml is:

scrape_configs:
# Scrape Prometheus itself every 2 seconds.
- job_name: 'prometheus'
  scrape_interval: 2s
  target_groups:
  - targets: ['localhost:9090', 'cadvisor:8080']

This file has path /home/test/prometheus.yml. To run the container with prometheus, I do:

docker run -d -p 42047:9090  --name=prometheus -v /home/test/prometheus.yml:/etc/prometheus/prometheus.yml  --link cadvisor:cadvisor prom/prometheus -config.file=/etc/prometheus/prometheus.yml -storage.local.path=/prometheus -storage.local.memory-chunks=10000

The container is created, but it dies immediately. Can you tell me where the problem is?

Messages form docker events& :

2016-11-21T11:43:04.922819454+01:00 container start 69d03c68525c5955cc40757dc973073403b13fdd41c7533f43b7238191088a25 (image=prom/prometheus, name=prometheus)
2016-11-21T11:43:05.152141981+01:00 container die 69d03c68525c5955cc40757dc973073403b13fdd41c7533f43b7238191088a25 (exitCode=1, image=prom/prometheus, name=prometheus)

score 9 · Answer 1 · edited Aug 03 '17 at 03:54

9

Config format is changed. targets come under static_config in the latest version.

scrape_configs:
# Scrape Prometheus itself every 2 seconds.
  - job_name: 'prometheus'
  scrape_interval: 2s
  static_configs:
      - targets: ['localhost:9090', 'cadvisor:8080']

Prometheus Documentation for further help

edited Aug 03 '17 at 03:54

Stephen Rauch

47,830
31
106
135

answered Aug 03 '17 at 03:28

Anon

129
1
9

score 7 · Answer 2 · answered Feb 10 '18 at 03:51

7

Yes, target_groups is renamed to static_configs. Please use the latest Prometheus image with the following.

static_configs:
  - targets: ['localhost:9090', 'cadvisor:8080']

The above worked for me.

answered Feb 10 '18 at 03:51

Vikram Hosakote

3,528
12
23

score 4 · Answer 3 · answered Dec 07 '16 at 13:06

I think target_groups have been deprecated from scrape_configs in latest version of prometheus. you can try static_configs or file_sd_config

scrape_config
static_config
file_sd_config

scrape_configs:
  - job_name: node_exporter
    static_configs:
      - targets:
         - "stg-elk-app-01:9100"
         - "stg-app-02:9100"

score 1 · Answer 4 · answered Nov 21 '16 at 10:15

1

The indentation isn't correct, try:

scrape_configs:
  # Scrape Prometheus itself every 2 seconds.
  - job_name: 'prometheus'
    scrape_interval: 2s
    target_groups:
    - targets: ['localhost:9090', 'cadvisor:8080']

answered Nov 21 '16 at 10:15

brian-brazil

31,678
6
93
86

2

Which version of Prometheus are you using? `target_groups` was renamed to `static_configs` a while back. This is very difficult to debug without any error output. – brian-brazil Nov 21 '16 at 10:31
I have done "docker pull prom/prometheus" , so I think it should be the latest (the TAG of the image is ''latest'') – SegFault Nov 21 '16 at 10:38
from logs: time="2016-11-21T11:21:40Z" level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): unknown fields in scrape_config: target_groups" source="main.go:149" – SegFault Nov 21 '16 at 11:24

score 1 · Answer 5 · answered Nov 20 '17 at 10:14

As you said in your earlier comment:

from logs: time="2016-11-21T11:21:40Z" level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): unknown fields in scrape_config: target_groups" source="main.go:149"

Which clearly means that the field "target_groups" is causing the problem. This is due to the fact that the new version of Prometheus (v1.5 onwards) have discarded the use of "target_groups" field and simply provide the targets. Even I faced this issue about 6 months ago. Please try it with a new version. The docker pull prom/prometheus might be getting you the old one.

Hope this helps...!!!

score 0 · Answer 6 · answered Aug 04 '17 at 19:18

The name of the container is prometheus.

Generally, when a container exists immediately after it starts, I would recommend adding the -log.level=debug right after -config.file.

docker run -d -p 42047:9090 --name=prometheus -v /home/test/prometheus.yml:/etc/prometheus/prometheus.yml --link cadvisor:cadvisor prom/prometheus -config.file=/etc/prometheus/prometheus.yml -log.level=debug -storage.local.path=/prometheus -storage.local.memory-chunks=10000

Next, see the logs for the container:

docker logs prometheus

Any issues with the configuration will be there.

Docker - Prometheus container dies immediately

6 Answers6