1

I was wondering if we don't explain any external_labels in Prometheus configuration file for Thanos, what will happen? for example I have 2 Prometheuses and they are scraping a same endpoint, so their external_labels is like this: prometheus1:

global:
  external_labels:
    replica: 1

prometheus2:

global:
  external_labels:
    replica: 2

at this point we know that 2 kind of metrics will be saved in the object storage, metrics with replica=1 label and replica=2 label.
if I delete the external_labels from configuration file, how many metrics will be saved in the object storage?

Hedeesa
  • 148
  • 1
  • 12

1 Answers1

2

I found the answer.
based on this doc,

Every Prometheus instance must have a globally unique set of identifying labels.

And if you don't declare one for your Prometheus, TSDB blocks wont be uploaded to your object storage and you gonna receive this error from sidecar:

level=warn ts=2020-10-03T17:00:14.826489487Z caller=sidecar.go:274 err="upload 01EKQPFYHW6R8TB48N07NSCH8P: empty external labels are not allowed for Thanos block." uploaded=0
Hedeesa
  • 148
  • 1
  • 12
  • Thanks for the info. I'm doing the same with 2 Prometheus instances scraping the same endpoint. I'm also using different replica names for both Prometheus. What could be the issue, any idea? – Harsha Apr 27 '23 at 09:32