Unable to add Grafana Loki datasource in Kubernetes

Question

I am following the following grafana documentation on loki. I am unable to properly connect to my k8s clusters loki logs after installing loki, promtail, and grafana via helm charts. When I enter the http: url in Add datasources within the Grafana GUI and proceed to save & test, grafana is unable to connect to loki.

My helm commands are:

helm upgrade --install --namespace=monitoring promtail grafana/promtail --set "loki.serviceName=loki"
helm upgrade --install loki --namespace=monitoring grafana/loki-distributed
helm install --namespace=monitoring loki-grafana grafana/grafana

Now I mainly am having trouble with this step and the syntax and how to debug the process: "using the URL http://helm-installation-name-gateway.namespace.svc.cluster.local/ for Loki (with and replaced by the installation and namespace, respectively, of your deployment)."

I have tried all of the following URLs with no luck, any guidance would be very appreciated!

http://loki-grafana.monitoring.svc.cluster.local:3100

Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details

Answer 1

I figured it out.

helm repo add loki https://grafana.github.io/loki/charts
helm repo update
kubectl create namespace monitoring
helm upgrade --install loki --namespace=monitoring grafana/loki-stack
helm upgrade --install grafana --namespace monitoring grafana/grafana
kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
kubectl port-forward --namespace loki service/grafana 3000:80

Then when you log into grafana use http://loki:3100 for the datasource url

The guide I found is here: https://medium.com/codex/setup-grafana-loki-on-local-k8s-cluster-minikube-90450e9896a8

I just skipped the minikube and ingress stuff

Answer 2

In general

This error:

Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details

Can occur due to multiple reason that are causing Grafana not not identify Loki as a datasource.
It is really depends on the use case and setup.

Use case

I'll answer on the following use case:

1. Loki's Open source version.
2. Running on K8S.
3. Grafana and Loki are running on the same cluster.
4. Loki is exposed with the Gatway component.

(*) I'm also using Promtail, but this is not related to the issue (saw some answers in the internet that suggested to debug there, no need - the failure is between Grafana and Loki gateway).

Verifying networking setup

Please see the 2 green rectangles below:

1. We're under K8S, so the HTTP url needs to follow the pattern of: <name-of-gateway-service>.<namespace>.svc.cluster.local. (You can try to connect with other values suggested under bullet (5) in the docs or directly to the pod, but I suggest sticking with the service pattern).
2. From here:

Loki defaults to running in multi-tenant mode. Multi-tenant mode is set in the configuration with auth_enabled: true. When configured with auth_enabled: false, Loki uses a single tenant. The X-Scope-OrgID header is not required in Loki API requests. The single tenant ID will be the string fake.

So, we need to add the under the X-Scope-OrgID header under the "Custom HTTP Headers" section or add this to the values.yaml file:

loki:
  auth_enabled: false

Other things that can go wrong

There are multiple reasons, especially if you're running in a non Monolithic mode, but I would suspect that if networking is OK, there might be an issue with the storage. Check the logs.