Stack Overflow
Stack Overflow

Questions tagged [prometheus]

The Prometheus monitoring system, including the server, alertmanager, push gateway, exporters, client libraries and other components.

Prometheus is a go-based open-source monitoring system inspired by Google's approach to monitoring.

Prometheus itself is a time-series storage server that periodically pulls metrics from monitored entities. These metrics can then be queried and alerted-upon using simple query and alert languages.

Prometheus comes with a suite of supporting tools like alertmanager, exporters and client libraries, designed to work together in providing a highly scalable and complete monitoring solution. Metric visualizations is usually done with Grafana, which seamlessly connects with Prometheus.

Prometheus's official site.

Read more about Prometheus's approach here:

  • This article explains Prometheus metric gathering approach for beginners.

  • This chapter from the online book "Site Reliability Engineering" describes Google's monitoring system Borgmon. Prometheus is mentioned in this chapter and was designed with Google's approach in mind.

6591 questions
24
votes
1 answer

Prometheus return no data when calculating a ratio of two metrics

I want to calculate a ratio of two metrics, but I get no data... I have some metrics like: fs_bytes{filesystem="/var",instance="localhost:9108",job="graphite",metric="Used"} …
Franklin Piat
  • 3,952
  • 3
  • 32
  • 45
24
votes
2 answers

Prometheus pre build binary for Mac OS X

I am trying out Prometheus on Mac OS X. I looked up the downloads and not having a direct indication of which version is for Mac. I tried docker to run the Prometheus on Mac. Just want to run it directly on Mac without docker. Does any one know…
Sam Tsai
  • 373
  • 1
  • 2
  • 15
24
votes
3 answers

Grafana Legend format :9100 removal

I have ":9100" showing up at the end of my metrics. I am using a prometheus datasource with my company and can't change it. Has anyone figured out how to remove this? I was thinking maybe templates or regex but I haven't been successful…
user3149615
  • 349
  • 1
  • 2
  • 5
23
votes
2 answers

multiple values from grafana variable in prometheus query

We have a situation where we need to select the multiple values (instances/servers) from grafana variable field, and multiple values needs to passed to the Prometheus query using some regex, so that i can see selected hosts metrics in single graph.…
Anil Kumar
  • 381
  • 1
  • 2
  • 9
23
votes
4 answers

How to monitor disk usage of persistent volumes?

I want to monitor disk usages of persistent volumes in the cluster. I am using CoreOS Kube Prometheus. A dashboard is trying to query with a metric called kubelet_volume_stats_capacity_bytes which is not available anymore with Kubernetes versions…
Cemal Unal
  • 514
  • 3
  • 6
  • 15
23
votes
3 answers

How to execute multiple queries in one call in Prometheus

I'm running prometheus inside kubernetes cluster. I need to send queries to Prometheus every minute, to gather information of many metrics from many containers. There are too match queries, so I must combine them. I know how I can ask Prometheus…
roie
  • 665
  • 1
  • 6
  • 14
23
votes
3 answers

Prometheus/PromQL subtract two gauge metrics

I have this gauge metric "metric_awesome" from two different instances. What i want to do, is subtract instance one from instance two like so metric_awesome{instance="one"} - metric_awesome{instance="two"} Unfortunately the result set is empty. Has…
alknows
  • 1,972
  • 3
  • 22
  • 26
23
votes
2 answers

Error from server (Forbidden): error when creating .. : clusterroles.rbac.authorization.k8s.io ...: attempt to grant extra privileges:

Failed to create clusterroles. <> already assigned as the roles of "container engine admin" & "container engine cluster admin" Error from server (Forbidden): error when creating…
ccyang2005
  • 281
  • 1
  • 3
  • 6
22
votes
2 answers

Simple cumulative increase in Prometheus

I have an application that increments a Prometheus counter when it receives a particular HTTP request. The application runs in Kubernetes, has multiple instances and redeploys multiple times a day. Using the query…
GavinE-OVO
  • 221
  • 2
  • 5
22
votes
3 answers

How to monitor Apache Spark with Prometheus?

I have read that Spark does not have Prometheus as one of the pre-packaged sinks. So I found this post on how to monitor Apache Spark with prometheus. But I found it difficult to understand and to success because I am beginner and this is a first…
xmlParser
  • 1,903
  • 4
  • 19
  • 48
21
votes
0 answers

Prometheus 2.x Limit Memory Usage

Basically I am looking for a equivalent of 1.x storage.local.target-heap-size. I have a prometheus app(running in a 6 core, 32G box) that pulls about host metrics from 2000 VMs. Scrape interval 5mins. Within few mins my RAM usage increases up to…
Subash Chaturanga
  • 814
  • 2
  • 10
  • 20
21
votes
5 answers

Use Prometheus operator with DB volume for k8s

We are trying to monitor K8S with Grafana and Prometheus Operator. Most of the metrics are working as expected and I was able to see the dashboard with the right value, our system contain 10 nodes with overall 500 pods. Now when I restarted…
JME
  • 881
  • 2
  • 11
  • 23
21
votes
1 answer

Prometheus Query Overall average under a time interval

How can I find the overall average of metrics over time interval ? avg(metric) = overall average value but avg_over_time(metrics[interval]) = averages value per label avg( avg_over_time(metric[scrape interval]) ) won't be same as(when the data is…
Vaisakh Rajagopal
  • 1,189
  • 1
  • 14
  • 23
21
votes
2 answers

Multiple email receivers in prometheus alertmanager "to" field

How can I create a receiver configuration with multiple email addresses in the "to" field?
eventhorizon
  • 2,977
  • 8
  • 33
  • 57
21
votes
3 answers

How to monitor disk usage of kubernetes persistent volumes?

I have container_fs_usage_bytes with prometheus to monitor container root fs, but it seems that there is no metrics for other volumes in cAdvisor.
hiroshi
  • 6,871
  • 3
  • 46
  • 59
1 2 3
…
99 100