Prometheus is not scraping stats d exporter

Question

I am extremely new to docker,k6, Prometheus,statds.

Scenario- Create a docker compost file which will run create

1. K6 load test which has custom metrics. Send the result to Statsd Exporter (The idea is to use Prometheus as the database but since K6 doesn't support directly support Prometheus, I am using Statsd. If anyone has a simpler solution than this please feel free to suggest.).
2. Prometheus which will scrape the results from statsd exporter.
3. Grafana - which will connect to Prometheus and show visualization.

But what happens is that Statsd metrics are generated including my custom metrics. But the Prometheus is not scraping it. The prometheus job is not getting listed in the Status->Targets page on prometheus.

Following is my docker-compose file.

version: '3.7'
services:
  statsdex:
    image: "prom/statsd-exporter:latest"
    container_name: statsd_exporter
    ports:
      - 9102:9102
    volumes:
            - ./loadtesting/statsd_mapping.yml:/tmp/statsd_mapping.yml 
  k6:
    image: "loadimpact/k6:0.32.0"
    command: ["run", "/scripts/K6-script.js","-q", "-o","statsd"]
    depends_on:
      - statsdex
    environment:
      K6_STATSD_ADDR: "statsdex:9125"
      K6_STATSD_ENABLE_TAGS: "true"
      K6_STATSD_PUSH_INTERVAL: 1s
      #K6_STATSD_BUFFER_SIZE: 7000
    volumes:
      - "./loadtesting:/scripts"
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_BASIC_ENABLED=false      
      - GF_INSTALL_PLUGINS=michaeldmoore-scatter-panel
    volumes:
      - grafana-storage:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/dashboards/
      - ./grafana/datasources:/etc/grafana/datasources/
      - ./grafana:/etc/grafana/provisioning/
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    ports:
        - 9090:9090
    volumes:
        - ./loadtesting/prometheus.yml:/etc/prometheus/prometheus.yml 

volumes:
  grafana-storage:
    external: true

Following is my Prometheus yaml file which I believe is trying to scrape the data from the same port where statds exporter is outputting results. (I don't see this job listed in the Prometheus. I tried giving the ip address, localhost too.)

scrape_configs:
  - job_name: 'statsd_exporter'
    static_configs:
      - targets: ['statsd_exporter:9102']

Following is the mapping file for statds exporter for K6 mapping.

defaults:
  observer_type: histogram
mappings:
- match: "k6.*"
  name: "k6_${1}"
- match: "k6.check.*.*.*"
  name: "k6_check"
  labels:
    http_name: "$1"
    check_name: "$2"
    outcome: "$3"

When I use the command docker compose up I can see

1. In the docker desktop that all the four services are up and running. K6 exits after the script execution is done.
2. I can see the Statsd Metrics at http://localhost:9102/metrics
3. I can see that Prometheus is set up at http://localhost:9090. But In the Status->Targets section I do not see Statsd Exporter. This is where I believe the problem is. Prometheus and Statsd exporter are not talking to each other. That may be the reason why I can not query the custom metrics which is in statsd exporter from Prometheus.

Following is the configuration in Prometheus.

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - follow_redirects: true
    scheme: http
    timeout: 10s
    api_version: v2
    static_configs:
    - targets: []
scrape_configs:
- job_name: prometheus
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  static_configs:
  - targets:
    - localhost:9090
- job_name: wmi_exporter
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  static_configs:
  - targets:
    - localhost:9182

Can anyone help me to figure out what exactly is the problem?

Answer 1

I figured out the problem and arrived at the solution.

prometheus and statsdex were not talking to each other. Prometheus was scraping itself instead of statsdex. I changed my docker compose for prometheus and gave the prometheus yml file details. I kept the prometheus file at the exact location given. so promethus reads from the config file.

prometheus: image: prom/prometheus command: - '--config.file=/etc/prometheus/prometheus.yml' container_name: prometheus ports: - 9090:9090
volumes:
- ./prometheus:/etc/prometheus - prometheus-data:/prometheus

In the prometheus yml file I gave the ip adderess instead of the name of the service.

scrape_configs:

• job_name: 'statsdex' metrics_path: /metrics
  static_configs:
  • targets: ['x.x.x.1:9102']

It worked like a charm.