When querying Datasource from grafana getting 403 for a few of the metrics.
curl --location --request GET 'https://xxx.xxx.xxx/api/datasources/proxy/1/api/v1/query?query=sum(kube_pod_container_status_restarts_total%7Bnamespace%3D%22default%22%2C%20container%3D~%22al-agent-container%22%2C%20pod%3D%22al-agent-container-hlrz2%22%7D)&time=1607489911' \
> --header 'Accept: application/json' \
> --header 'Content-Type: application/json; charset=UTF-8' \
> --header 'Authorization: Bearer xxx' \
> --data-raw ''
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
</body>
</html>
When Querying from the Prometheus server endpoint with same query it resolves with 200:
curl --location --request GET 'http://kube-system-prometheus-server.kube-system.svc.cluster.local/api/v1/query_range?query=sum(kube_pod_container_resource_requests_cpu_cores%7Bnamespace%3D%22default%22%2C%20pod%3D~%22al-agent-container-hlrz2%22%7D)&start=1607488035&end=1607489835&step=15' \
> --header 'Accept: application/json' \
> --header 'Content-Type: application/json; charset=UTF-8'
{"status":"success","data":{"resultType":"matrix","result":[]}}
Grafana Version: v6.6.2 (3fa63cfc34)
Prometheus does not have any authentication in place, This issue is facing only for few metrics not for all.
When tested the data source from its configuration section. it shows it's all working.
does anyone have any idea on this?
