Stack Overflow
Stack Overflow

Questions tagged [thanos]

55 questions
1
vote
2 answers

Grafana filtering Thanos store labels| Grafana Promtheus Thanos

I was wondering is there any way to filter by Thanos store in Thanos+prometheus+Grafana cluster. I am trying to query from Grafana dashboard to get results from exactly one store (we have 3). Any help is appreciated, Thank you in advance
Behzod Olloqulov
  • 11
  • 4
1
vote
0 answers

gRPC connection between two different meshes is reset

I have two different clusters (EKS, v1.18) with their own meshes (v1.9.0). I have a Thanos deployment on cluster A and a Prometheus deployment on cluster B (with the thanos sidecar running too). The goal is to have thanos query these sidecars in…
Kevin Mai
  • 65
  • 7
1
vote
0 answers

Thanos Receiver Data Retention for 3 hours

I did set up 5 replicas of Thanos Receivers with 3 hours of data retention. I verified that ULID folders created every 2 hours are being created and being deleted after 3 hours. However, my problem is, the data in the wal directory seems keep on…
Yejin
  • 541
  • 2
  • 15
  • 32
1
vote
1 answer

what if we don't explain any external_labels in Prometheus for Thanos

I was wondering if we don't explain any external_labels in Prometheus configuration file for Thanos, what will happen? for example I have 2 Prometheuses and they are scraping a same endpoint, so their external_labels is like…
Hedeesa
  • 148
  • 1
  • 12
1
vote
2 answers

Thanos S3 Config

I am struggling to get my head around of using S3 bucket for thanos. It appears from the config that we have to provide access and secret keys for the S3 bucket, but they are being written in plain sight and as the code is stored in git so anyone…
sjm
  • 415
  • 1
  • 9
  • 23
1
vote
1 answer

Thanos Querier can only query data 2 hours ago

I followed this blog https://rancher.com/blog/2020/prometheus-metric-federation in setting up thanos in our kubernetes. We have this set up where Cluster A contains only the - Thanos Receiver, Thanos Store and Thanos Query. The data chunks are…
Yejin
  • 541
  • 2
  • 15
  • 32
1
vote
1 answer

Remote write directly to Thanos

Could you say please if it is possible to write directly to Thanos (without Prometheus) f.e. by means of REST API? Can't find any example.
Альберт Александров
  • 807
  • 1
  • 15
  • 27
0
votes
0 answers

Empty result when calculating erro rate ratio

I need to compare the error rate of two different HTTP client versions with that of a legacy client, for each possibile combination of status code and client version. As an example of what I'm trying to achieve, consider that I have the following…
c0tonet
  • 23
  • 6
0
votes
1 answer

Exposing grpc server with traefik ingress on kubernetes cluster

I'm trying to make a gRPC service (thanos sidecar) externally accessible over a domain in my kubernetes cluster (k3s cluster). I am using Traefik as an ingress controller. Any clues as to what I may be misconfiguring would be much appreciated. I am…
Beefcake
  • 305
  • 4
  • 10
0
votes
1 answer

How do i set response_header_timeout in Thanos Query Frontend Manifest?

I want to increase the response_header_timeout in thanos query-frontend deployment manifest as my queries are timing out, but can't find the correct syntax format expected. Current Syntax I am following: - -| …
Shrey Makwana
  • 1
  • 1
0
votes
0 answers

Summary of metrics collected on Prometheus

I collect metrics from systems with node exporter or from applications' own endpoints. Some of these are ever-increasing metrics like sales volume (counter), and some are metrics that change (gauge) like CPU load. I collect metrics with 400…
OmerFaruk
  • 48
  • 5
0
votes
0 answers

How to deal with potential clock skew from incoming Prometheus remote_write streams Thanos

I encountered an issue that more or less took Thanos down. I spun up one of my internal "observability stacks" about 2PM on Monday. I was feeding my stack with Grafana agent running on each of a few "instrumented machines". About 5:30PM, I spun…
Frank Siler
  • 21
  • 5
0
votes
0 answers

With Helidon Metrics how do I mimic the OEM metrics?

With the metrics available in Helidon, how do I make graphs in Grafana to mimic what we used to have in OEM? For example: We seem to have Microprofile metrics, such as "base_gc_time_seconds" but I guess we need to use rate or something? Thanks!
Sarkie
  • 272
  • 3
  • 18
0
votes
0 answers

Thanos link is missing in the source when the alert triggers in Prometheus stack v43.2.0

Thanos link is missing in the **source ** when the alerts are triggered in Opsgenie. Example: Source:…
Akheel Mohammed
  • 1
  • 1
0
votes
1 answer

Why KSM (Kube-State-Metrics) is being scraped by only one Prometheus shard?

We are deploying Prometheus with sharding capabilities using thanos sidecar. Prometheus has the below recording rule: sum by (cluster, namespace, pod, container) ( irate(container_cpu_usage_seconds_total{job="kubelet",…
Haitham00n
  • 31
  • 4
1
2
3 4