Environment variables in prometheus values

Question

i would like to set environment specific values based on environment qa/prod in prometheus values file

 ## Additional alertmanager container environment variable
  ## For instance to add a http_proxy
  ##
  extraEnv: {}

Answer 1

Prometheus does not support environment variables. There are a few threads discussing this in GitHub.

You can use envsubs or if you want a more robust and versatile tool I recommend to GO with Confd (its written in GO :D). You can also get the secrets from backends like etcd or AWS ssm.

Here you have Pormetheus with confd ready to go (just need to modify the config to your needs.

Answer 2

Prometheus doesn't support environment variables in config file. But other Prometheus-like systems such as VictoriaMetrics support env vars in Prometheus-compatible config files via %{ENV_VAR} syntax. For example, the following config would substitute %{ENV} with qa or prod if the corresponding ENV=qa or ENV=prod environment variable is passed to VictoriaMetrics or vmagent:

scrape_configs:
- job_name: foo
  static_configs:
  - targets: [foobar:1234]
    labels:
      env: '%{ENV}'

This allows using the same Prometheus config in different environments and templating it with environment variables. See these docs for details.