1

I followed this blog https://rancher.com/blog/2020/prometheus-metric-federation in setting up thanos in our kubernetes.

We have this set up where Cluster A contains only the - Thanos Receiver, Thanos Store and Thanos Query.

The data chunks are successfully loaded in the S3 every 2 hours. However, when we pointed the Grafana Dashboard to use the Thanos Query, no data results are shown if you query in range of 10 minutes (or less than 2 hours). But querying data with time range 2 hours ago and up, data are shown.

Is Thanos Querier can query real time data from the Thanos Receiver (from Prometheus)? This is my configuration for Thanos Querier deployment (below) Is there any configuration I have missed?

containers:
      - args:
        - query
        - --log.level=info
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:9090
        - --query.replica-label=prometheus_replica
        - --query.replica-label=rule_replica
        - --store=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local
        image: quay.io/thanos/thanos:v0.15.0

enter image description here

Yejin
  • 541
  • 2
  • 15
  • 32

1 Answers1

1

Yes, Thanos Query is able to get real-time data from Thanos Receive. Thanos documentation for receive claims:

It exposes the StoreAPI so that Thanos Queriers can query received metrics in real-time.

Just add another --store option to your query invocation to point to thanos-receive gRPC port

bjakubski
  • 1,477
  • 10
  • 9
  • Hi Sir @bjakubski, thanks for the reply. I updated my config and added the ff: - --store=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local - --store=thanos-store:10901. However, there is still no results in the querier when querying data real time. Message appears "No StoreAPIs matched for this query" and "No Data points found". Even the status is Up and Healthy. – Yejin Sep 23 '20 at 02:32
  • Seems you are pointing to thanos-store again (judging by the hostname used) Make sure you are using the receiver there! If you have access to Thanos Query UI then you can check the "Stores" tab to see the status of stores you've configured. – bjakubski Sep 23 '20 at 08:49
  • Sir @bjakubski, thank you for the help. I have added the receiver in my store config, and I can see their status in the "Stores" page. I am able to query "count (up) by (tenant_id)" in Thanos Query and data results were shown. You have saved my day. Thank you so much Sir. – Yejin Sep 24 '20 at 07:19