17

I have a new server running Prometheus in docker-compose. I want to be able to re-load the configuration file (prometheus.yml) without have to stop and start the container.

Of course since I persist the storage of promethues in a volume the stop and start isn't really a problem but it seems like overkill, especially since prometheus itself has such a handy api to reload configs.

I see other people with similar questions (e.g. here) but I have been unable to get those solutions to work for me. Maybe I'm overlooking something there.

docker-compose.yml

version: "3"

services:

  grafana:
    restart: always
    container_name: grafana
    image: grafana/grafana:6.2.1
    ports:
      - 3000:3000
    volumes:
      - grafanadata:/var/lib/grafana

  prometheus:
    restart: always
    container_name: prometheus
    image: prom/prometheus:v2.10.0
    privileged: true
    volumes:
      - ./configuration/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheusdata:/prometheus

    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.enable-admin-api'
      - '--web.enable-lifecycle'
    ports:
      - 9090:9090

  node:
    restart: always
    container_name: node
    image: prom/node-exporter:v0.18.0
    ports:
      - 9100:9100

volumes:
  grafanadata:
  prometheusdata:

Alas, my results..

When I run curl -X POST http://localhost:9090/-/reload the docker-compose logs give:

prometheus    | level=info ts=2019-06-17T15:33:02.690Z caller=main.go:730 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
prometheus    | level=info ts=2019-06-17T15:33:02.691Z caller=main.go:758 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml

So the prometheus' end is working fine.. All good so far.

However, when I edit ./configuration/prometheus/prometheus.yml the changes don't propogate to the container. Furthermore, when I try to edit /etc/promethus/prometheus.yml in container I see that it is read only (and as an aside, the container does not have a 'sudo' command).

Is there a docker native way to hot/live reload these config files to the container directory?

As stated, the down/start option works for now but I'm curious if there is a more elegant solution.

goldfishalpha
  • 447
  • 2
  • 4
  • 13

5 Answers5

11

docker-compose kill -s SIGHUP prometheus does the trick, so Vishrant was certainly on to something there.

codehead
  • 2,077
  • 16
  • 20
4

I will assume you have already updated the /etc/prometheus/prometheus.yml file and you are just asking about the reloading of config. (Though the config can be part of configmap, and you can avoid the pod restart when the configmap changes).

Run the following command in order to reload the config without stopping the prometheus process:

docker exec <prometheus_container_name> sudo killall -HUP prometheus

Explanation: sudo killall -HUP prometheus command sends a SIGHUP signal to the prometheus process to reload the configurations.

Validate the changes by calling:

curl -X GET http://localhost:9090/api/v1/status/config
Vishrant
  • 15,456
  • 11
  • 71
  • 120
3

Your problem might be, that your editor is not saving the changed file under the same inode. So dockerd does not realize that the file changed, since its written to a new inode. Solutions might be to mount the complete directory or tell your editor to not use a temporary file:

I.e. for Sublime, setting "atomic_save": false to user preferences might help (results unclear) or for vim see https://github.com/moby/moby/issues/15793#issuecomment-571932545

This issue is very interesting for this topic in general: https://github.com/moby/moby/issues/15793

oxivanisher
  • 46
  • 4
  • Enlightening! Thank you for posting on this old-ish post. It may be some months until I get the opportunity to revisit this, but I will update here once I do. – goldfishalpha May 12 '20 at 12:21
1

I've got a slightly different docker-compose.yml file. I use the following script to reload prometheus:

#!/bin/bash

promtool='docker-compose exec prometheus promtool'

error_found=0

${promtool} check config /etc/prometheus/prometheus.yml || error_found=1

# See mapping in docker-compose
# ./config/:/config/ allow us a local lookup and remote check
# of rules files
for rules_file in ./config/alerts/*yml
do
    ${promtool} check rules "/${rules_file}" || error_found=1
done

if [[ ${error_found} -gt 0 ]]; then
    echo "Not reloading config, errors found."
else
    docker-compose kill -s SIGHUP prometheus
fi

It will check the config and rules files first. If they are fine prometheus will be triggered to reload.

Marco
  • 11
  • 1
-1

It looks like

      - ./configuration/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml

should be reversed.

saalih123123
  • 33
  • 3