0

I have a prometheus server querying a number of hosts and running a thanos-sidecar configured against s3 bucket. Metrics are pushed without errors, on my test s3 bucket i see them earliest mid of september.

Thanos-server, which is a separate machine is running:

  • grafana (default source - thanos-query localhost:29090)
  • thanos-store (its able to read the s3 bucket)
  • thanos-query (pointed to thanos-store)
  • thanos-compactor as cronjob (just in case it matters: --retention.resolution-raw=3d --retention.resolution-5m=14d --retention.resolution-1h=90d)

I have set prometheus retention to 3days. Which should be the value indicating when prometheus clears up local storage.

When i make a simple test with uptime node-exporter statistics over 30days, shows just last 3 days of history. Grafana is querying thanos-query which should pull down the metrics via thanos-store from the s3 bucket and show me all data on the bucket

I most likely missed a piece of configuration in the stack.

strzelecki.maciek
  • 324
  • 2
  • 22
  • Could you share configs for Thanos-query and Thanos-store? It looks like there are some issues with these configs, which prevent from reading data from s3. I'd also recommend trying other long-term remote storage solutions for Prometheus such as M3SB, Cortex or [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/), since they may be easier to configure and operate comparing to Thanos. – valyala Nov 09 '20 at 18:45
  • There is very little of configuration. In thanos query i can see the local s3-connected store (in addition to all remote stores from thanos-sideras), with all labels, time-min going back months of history. Yet all grafana queries are just showing last 3 (thats prometheus retention) days obtained via thanos-sidecar. Is there any special configuration needed to make thanos actually USE the long term s3 storage? – strzelecki.maciek Nov 23 '20 at 00:01
  • Short answer for your question in the comment is that you don't need any specific config. Thanos Querier should query metrics from S3 via Thanos Store Gateway and Grafana should show all the metrics. I suspect that the issue you are facing is a configuration issue in Thanos Store Gateway, so it is not getting metrics from S3. Hard to pinpoint the issue without any screenshots or config details. Add some screenshots if you are still interested to resolve this issue. – shan1024 Mar 14 '21 at 17:32

0 Answers0