How do I configure the log level of Prometheus' Alertmanager?

Question

I've set up Prometheus' Alertmanager to, well, manage alerts from Prometheus. I've got the alerts coming in from Prometheus to Alertmanager, but there the story ends. The alerts never get sent on by Alertmanager to my email endpoint.

In order to figure out where exactly inside Alertmanager that the alerts end their journey I want to turn the log level from info to debug, but have been unable to figure out how. Even finding the log seems like a tough ask right now, its not in /var/log and journalctl -u alertmanager contains so little that it may be that there is another log somewhere.

The manual page for configuring Alertmanager does not mention debug level. I've looked through the source code for mentions of log and found that the setting should be named log.level. Adding the following snippet to the configuration YAML didn't help either:

log:
  level: debug

as Alertmanager failed to start with a failure to parse its config file.

score 4 · Accepted Answer · answered Dec 15 '20 at 10:56

The answer is that its not possible to set Prometheus' Alertmanagers log level to debug through the config file, it is only possible through commandline arguments. Do not ask me why, I'm sure they had their reasons.

Through Puppet I added the argument to the Systemd unit file for Alertmanager, so that it ended up looking like this:

[Unit]
Description=Prometheus alertmanager
Wants=basic.target
After=basic.target network.target

[Service]
User=alertmanager
Group=alertmanager
EnvironmentFile=/etc/sysconfig/alertmanager
ExecStart=/usr/local/bin/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yaml \
--storage.path=/var/lib/alertmanager \
--log.level=debug

ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target

If you are starting Alertmanager from your shell you can just add the flag --log.level=debug to your invocation.

The debug messages can then be seen via journalctl -u alertmanager on Linux distributions with the Systemd init system.

Vsevolod Gromov · Answer 2 · 2022-01-17T12:27:35.123

3

On Ubuntu systems (if prometheus-alertmanager was installed via apt and from official repo).

Open

/etc/default/prometheus-alertmanager

And add

ARGS="--log.level=debug"

to file (replace ARGS="")

Then restart prometheus-alertmanager via systemd:

sudo systemctl restart prometheus-alertmanager.service

Show logs via journalctl:

journalctl -u prometheus-alertmanager.service --since today

edited Jan 17 '22 at 12:27

answered Jan 14 '22 at 13:24

Vsevolod Gromov

141
3

score 0 · Answer 3 · answered Nov 15 '22 at 15:15

In my docker-compose.yml I did next: command: --log.level=debug

See details:

  alertmanager:
    restart: unless-stopped
    image: prom/alertmanager:latest
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
      - '--log.level=debug'
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    expose:
      - 9093/tcp
    networks:
      monitoring-net:

How do I configure the log level of Prometheus' Alertmanager?

3 Answers3