How can I use an ngnix proxy to scrape Prometheus metrics using a custom HTTP header?

Question

I need to scrape Prometheus metrics from an endpoint that requires a custom HTTP header, x-service-token.

Prometheus does not include an option to scrape using a custom HTTP header, only the Authorization header.

One user shared a workaround for using nginx to create a reverse proxy

Just in case others come looking here for how to do this (there are at least 2 other issues on it), I've got a little nginx config that works. I'm not an nginx expert so don't mock! ;)

I run it in docker. A forward proxy config file for nginx listening on 9191:

http {
  map $request $targetport {
    ~^GET\ http://.*:([^/]*)/ "$1";
  }
  server {
    listen 0.0.0.0:9191;
    location / {
      proxy_redirect off;
      proxy_set_header NEW-HEADER-HERE "VALUE";
      proxy_pass  $scheme://$host:$targetport$request_uri;
    }
  }
}
events {
}

Run the transparent forward proxy:

docker run -d --name=nginx --net=host -v /path/to/nginx.conf:/etc/nginx/nginx.conf:ro nginx

In your prometheus job (or global) add the proxy_url key

  - job_name: 'somejob'
    metrics_path: '/something/here'
    proxy_url: 'http://proxyip:9191'
    scheme: 'http'
    static_configs:
    - targets:
      - '10.1.3.31:2004'
      - '10.1.3.31:2005'

Originally posted by @sra in https://github.com/prometheus/prometheus/issues/1724#issuecomment-282418757

I have tried configuring this, but without 'host' networking and using host.docker.internal instead of localhost, but nginx is not able to connect

nginx               | 172.26.0.4 - - [31/Oct/2022:16:07:38 +0000] "GET http://host.docker.internal:8080/actuator/prometheus HTTP/1.1" 502 157 "-" "Prometheus/2.39.1"

This workaround also requires saving the API key in a file, which is not ideal, as this could accidentally be committed to a repo.

Prometheus locked the GitHub issue, so users are not able to ask for help or follow up questions.

There are two other StackOverflow questions on this topic, but the answers do not attempt to provide workarounds:

• Prometheus scrape /metric with custom header
• Adding custom header in HTTP request of prometheus

Answer 1

I've managed to get this working with an nginx proxy that runs on the same Docker network as the Prometheus instance.

.
├── config/
│   ├── nginx.conf
│   └── prometheus.yml
└── docker-compose.yml

Prometheus is configured to scrape the Prometheus metrics from nginx.

URLs

I have 3 environments, 'local', 'dev', and 'prod'.

The Prometheus metrics are available at the following URLs. Note that dev and prod require HTTPS and an API key, but local does not.

• local - http://localhost:8080/metrics/prometheus
• dev - https://dev.my-app.website.com/metrics/prometheus
• prod - http://prod.my-app.website.com/metrics/prometheus

nginx config

The nginx server has been configured to forward the requests to each environment based on the port.

• :9191 - local
• :9192 - dev
• :9193 - prod

I have manually defined the URLs for each environment in each nginx server { } block (except for 'localhost'), because nginx or Prometheus doesn't seem to like resolving the correct URL otherwise. It's a mystery.

http {

  resolver 127.0.0.11 ipv6=off; # use the docker DNS, to resolve host.docker.internal

  map $request $target_port {
    ~^GET\ http://.*:([^/]*)/ "$1";
  }

  # local
  server {
    listen 9191;

    location / {
      # no need for API key on local env
      # proxy_set_header x-api-key ...;
      proxy_set_header Host localhost;
      proxy_pass http://$host:$target_port$request_uri;
    }
  }

  # dev
  server {
    listen 9192;

    location / {
      proxy_set_header x-api-key DEV_API_KEY_123_ABC;
      proxy_set_header Host dev.my-app.website.com;
      proxy_pass https://dev.my-app.website.com:443$request_uri;
    }
  }

  # prod
  server {
    listen 9193;

    location / {
      proxy_set_header x-api-key PROD_API_KEY_999_XYZ;
      proxy_set_header Host prod.my-app.website.com;
      proxy_pass https://prod.my-app.website.com:443$request_uri;
    }
  }
}
events {
}

Prometheus config

Prometheus is configured to use the nginx container as a proxy URL.

Because nginx and Prometheus are running in the same Docker network, I can specify nginx by the container name.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:

  - job_name: "my-backend-local"
    proxy_url: "http://nginx:9191"
    metrics_path: "/monitor/prometheus"
    scrape_interval: 2s
    static_configs:
      - targets: [ "host.docker.internal:6060" ]
        labels:
          application: "my-backend"
          env: "local"

  - job_name: "my-backend-dev"
    proxy_url: "http://nginx:9192"
    metrics_path: "/monitor/prometheus"
    scrape_interval: 2s
    static_configs:
      - targets: [ "dev.my-app.website.com" ]
        labels:
          application: "my-backend"
          env: "dev"

  - job_name: "my-backend-prod"
    proxy_url: "http://nginx:9193"
    metrics_path: "/monitor/prometheus"
    scrape_interval: 2s
    static_configs:
      - targets: [ "prod.my-app.website.com" ]
        labels:
          application: "my-backend"
          env: "prod"

Docker Compose config

Finally, the Prometheus and nginx Docker instances are configured to read the ./config/prometheus.yml and ./config/nginx.conf files.

version: "3.9"

services:

  prometheus:
    image: prom/prometheus:v2.39.1
    container_name: prometheus
    volumes:
      - "./config/prometheus.yml:/etc/prometheus/prometheus.yml"
      - "./data/prometheus:/prometheus"
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/etc/prometheus/console_libraries"
      - "--web.console.templates=/etc/prometheus/consoles"
      - "--web.enable-lifecycle"
    ports:
      - "9090:9090"

  backend-proxy:
    image: nginx
    container_name: nginx
    restart: unless-stopped
    volumes:
      - "./config/nginx.conf:/etc/nginx/nginx.conf:ro"

Answer 2

I brought you a complete setup with an app, a forward proxy, and prometheus in docker-compose. It's quite long, so I'm putting it after the explanation. Please note that, just as with your solution, it does not work with host.docker.internal as it seems that NGINX does not use /etc/hosts when resolving hosts: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/259#issuecomment-1125197753 . All other hosts should work fine and you can use host's IP address instead of host.docker.internal if you need so.

You can run this by saving the contents into docker-compose.yml and running docker-compose up from the same directory. After 10 seconds or so you should see in logs how requests go through the proxy to the app and the app will show you the headers that it got. You can then proceed to Prometheus UI (localhost:9090) and query for metric the_answer_is to further check that everything is in place.

The proxy works as following:

• the target param (e.g. GET /?target=example.com) is the host or IP where the actual metrics are, this is the only mandatory parameter;
• if there is a scheme param, use it as the protocol, default - "http";
• if there is a host param, use it as Host header, default - the value of target param;
• if there is a port param, use it as a TCP port, default is "" (determined by the scheme);
• if there is a secret_token param, it gets injected into X-Custom-Header, default is "";

I recommend testing the proxy with curl, like this:

curl 'localhost/something/here?target=someapp&port=8000&secret_token=foo

Now goes the docker-compose.yml:

version: '3'
networks:
  app: # a network where the app is
  no_app: # a network where there is no app so that prometheus can't reach it directly

services:
  # A basic http server that exposes one metric and prints some headers along the way
  someapp:
   image: tiangolo/uwsgi-nginx-flask:python3.8-alpine
   entrypoint: ["/usr/local/bin/python", "-c"]
   networks:
    - app
   ports:
     - 8000:8000
   command:
   - |-
     import json
     from flask import Flask, request
     app = Flask(__name__)

     import logging
     log = logging.getLogger('werkzeug')

     @app.route('/', defaults={'path': ''})
     @app.route('/<path:path>')
     def print_request(path):
         log.info(f"{'-'*78}\n{str(request.headers).strip()}")
         if request.path == "/something/here":
           return "the_answer_is 42\n"
         else:
           return "OK\n"

     app.run("0.0.0.0", port=8000)

  # A forward proxy for Prometheus
  forward_proxy:
    image: nginx
    networks:
      - app
      - no_app
    ports:
      - 80:80
    entrypoint: ["/bin/bash", "-c"]
    environment:
      # Note that single "$" is considered by docker-compose as its variable, 
      # double "$$" is just an escape here
      config: |
        # Default scheme
        map $$arg_scheme $$target_scheme {
          ~.+     $$arg_scheme;
          default http;
        }
        # Default host (header) from target
        map $$arg_host $$target_host {
          ~.+     $$arg_host;
          default $$arg_target;
        }
        # This is to add ":" between target ip or host and port
        map $$arg_port $$has_port {
          ~.+     ":";
          default "";
        }

        # use docker internal DNS to resolve "someapp"
        resolver 127.0.0.11 ipv6=off;

        server {
          listen 80;

          location / {
            proxy_set_header Host $$target_host;
            proxy_set_header X-Custom-Header $$arg_secret_token;
            proxy_pass $$target_scheme://$$arg_target$$has_port$$arg_port$$request_uri;
          }
        }

    command:
    - |-
      set -euo pipefail
      echo -e "$$config" >/etc/nginx/conf.d/default.conf
      echo -e "==== NGINX Config ====\n$$(cat /etc/nginx/conf.d/default.conf)"
      nginx -g 'daemon off;'

  prometheus:
    image: prom/prometheus:v2.29.2
    entrypoint: ["/bin/sh", "-c"]
    ports:
      - 9090:9090
    command:
    - |
      echo -e "$$config" > /etc/prometheus/prometheus.yml
      echo -e "==== Prometheus Config ====\n$$(cat /etc/prometheus/prometheus.yml)"
      /bin/prometheus --config.file=/etc/prometheus/prometheus.yml \
                      --storage.tsdb.path=/prometheus \
                      --web.console.libraries=/usr/share/prometheus/console_libraries \
                      --web.console.templates=/usr/share/prometheus/consoles
    networks:
      - no_app
    environment:
      config: |
        scrape_configs:
        - job_name: 'somejob'
          scrape_interval: 10s
          metrics_path: '/something/here'

          # set params for the proxy ($$arg_NAME)
          params:
            port: ["8000"]
            secret_token: ["foo"]  # beware, this will be visible in Prometheus UI under "config" section

          static_configs:
          - targets:
            - 'someapp'
          
          # Here we replace actual target with the address and port of our forward_proxy
          # If you're familiar with it, this is exactly the same as for blackbox exporter
          relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: forward_proxy:80  # The forward proxy address and port