This is how 2 services among many in consul look like:
"1": {
"ID": "1",
"Service": "Service X",
"Meta": {
"country": "USA"
},
"Port": 0,
"Address": "1.1.1.1",
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false
}
"2": {
"ID": "2",
"Service": "Service Y",
"Meta": {
"country": "CANADA"
},
"Port": 0,
"Address": "2.2.2.2",
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false
}
Each of these services has health checks.
My goal is to find all healthy services based on consul KV. Like, find all services that are "passing" in "USA", or find all "critical" services in "CANADA".
I have tried a few options, the problem I get into is that all metrics have each relabel that is mentioned in config.yaml
QUERY 1
consul_health_service_status{status="passing", instance="USA"}
OUTPUT
consul_health_service_status{check="service:1",instance="USA",job="consul",node="machine",service_id="1",service_name="Service X",status="passing"} 1
consul_health_service_status{check="service:2",instance="USA",job="consul",node="machine",service_id="2",service_name="Service Y",status="passing"} 1
QUERY 2
consul_health_service_status{status="passing", instance="CANADA"}
OUTPUT
consul_health_service_status{check="service:1",instance="CANADA",job="consul",node="machine",service_id="1",service_name="Service X",status="passing"} 1
consul_health_service_status{check="service:2",instance="CANADA",job="consul",node="machine",service_id="2",service_name="Service Y",status="passing"} 1
CONFIG
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'consul'
consul_sd_configs:
- server: 'localhost:8500'
services: []
relabel_configs:
- source_labels: [__meta_consul_service_metadata_country]
target_label: instance
- source_labels: ['__meta_consul_address']
separator: ';'
target_label: '__address__'
replacement: 'localhost:9107'
action: 'replace'
I would loved to see an output like:
QUERY 1
consul_health_service_status{status="passing", instance="USA"}
OUTPUT
consul_health_service_status{check="service:1",instance="USA",job="consul",node="machine",service_id="1",service_name="Service X",status="passing"} 1
QUERY 2
consul_health_service_status{status="passing", instance="CANADA"}
OUTPUT
consul_health_service_status{check="service:2",instance="CANADA",job="consul",node="machine",service_id="2",service_name="Service Y",status="passing"} 1