Accessing the Grafana API through k8s proxy

Question

I am running Grafana v6.2.4 in kubernetes, using basic auth. I want to use the k8s proxy for testing (i.e. kubectl proxy --port=8080). I have changed the GF_SERVER_ROOT_URL environment variable to:

{
    "name": "GF_SERVER_ROOT_URL",
    "value": "http://localhost:8080/api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/"
}

This allows me to log in and use Grafana through my browser at http://localhost:8080/api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/.

However, I want to use it via the API. If I send a request to http://localhost:8080/api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/api/dashboards/db I get back

{
    "message": "Unauthorized"
}

However, if I set up a kubernetes port forward and send the identical request to http://localhost:30099/api/dashboards/db, then it succeeds.

Is there a different environment variable aside from GF_SERVER_ROOT_URL that I should be changing so that the API server root goes through the k8s proxy, i.e. http://localhost:8080/api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/api/dashboards/db? I have looked here but couldn't find it.

Otherwise what is the correct way to access the API through the k8s proxy?

I should add that I am specifically trying to use kubetctl proxy as an alternative to kubectl port-forward so I'm hoping to find an alternative to the suggestion here https://stackoverflow.com/a/45189081/1011724

Basic auth, using postman. I used the same method for both the proxy and the port forward and the port forward authentication works. — Dan, Sep 02 '19 at 19:44
My guess: something modifies `Authorization` request header (maybe adds Bearer token), so Grafana doesn't recognize it. Try to sniff request in Grafana pod and check request headers. — Jan Garaj, Sep 02 '19 at 19:48

mikejoh · Accepted Answer · 2019-09-02T20:15:07.397

I tried to replicate this in minikube and i might have found the reason for your requests through the API server proxy (using kubectl proxy) not getting authorized correctly.

Running the following curl-command:

curl -H "Authorization: Bearer <TOKEN>" http://localhost:8080/api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/api/dashboards/home

and using tcpdump to capture the requests in the Pod with tcpdump -vvvs 0 -l -A -i any yielded the following result(s):

GET /api/dashboards/home HTTP/1.1                                                                            
Host: localhost:8080                                                                                                 
User-Agent: curl/7.58.0                                                                                              
Accept: */*                                                                                                          
Accept-Encoding: gzip                                                                                                
X-Forwarded-For: 127.0.0.1, 192.168.99.1                                                                             
X-Forwarded-Uri: /api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/api/dashboards/home

This GET request doesn't have the Authorization header (resulting in a 401 Unauthorized) so basically the API server seem to strip this HTTP header as it passes the request down to the Pod.

If i instead use kubectl port-forward -n my-namespace svc/grafana-prom 8080:80 the GET request now looks like this:

GET /api/dashboards/home HTTP/1.1                                                                            
Host: localhost:8080                                                                                                 
User-Agent: curl/7.58.0                                                                                              
Accept: */*                                                                                                          
Authorization: Bearer <TOKEN>

When writing this answer i found this issue in the k/k repo #38775, to qoute one of the comments:

this is working as expected. "proxying" through the apiserver will not get you standard proxy behavior (preserving Authorization headers end-to-end), because the API is not being used as a standard proxy

This basically means that kubectl proxy will not work when trying to authorize through it, it's not a "regular" reverse proxy and probably for a good reason won't preserve Authorization headers.

Note that i tested both token and basic authorization using curl although token based auth is used above.

Hopefully this clear things up a bit!

Thanks, I'm not sure I understood all that. But while I read up on it, is the bottom line that I can't use the Grafana API through the kubectl proxy? — Dan, Sep 02 '19 at 20:01
Also, I was using basic auth, not a token, in case that changes anything. — Dan, Sep 02 '19 at 20:01
Sorry about the confusion regarding the token, i tested this with `Authorization: Basic` header using `curl` and i got the same result. And yes, the bottom line is that you can't use `kubectl proxy` as intended since the `Authorization` headers are stripped. As a sidenote, sending any other arbitrary HTTP headers are kept as-is when passing through the API server proxy. I'll update the answer. — mikejoh, Sep 02 '19 at 20:11

Accessing the Grafana API through k8s proxy

1 Answers1