How to access the prometheus & grafana via Istion ingress gateway? I have installed the promethius anfd grafana through Helm

Question

I used below command to bring up the pod:

kubectl create deployment grafana --image=docker.io/grafana/grafana:5.4.3 -n monitoring

Then I used below command to create custerIp:

kubectl expose deployment grafana --type=ClusterIP --port=80 --target-port=3000 --protocol=TCP -n monitoring

Then I have used below virtual service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana
spec:
  hosts:
  - "*"
  gateways:
  - cogtiler-gateway.skydeck
  http:
  - match:
    - uri:
        prefix: /grafana
    route:
    - destination:
        port:
          number: 3000
        host: grafana

kubectl apply -f grafana-virtualservice.yaml -n monitoring

Output:

virtualservice.networking.istio.io/grafana created

Now, when I try to access it, I get below error from grafana:

 **If you're seeing this Grafana has failed to load its application files

 1. This could be caused by your reverse proxy settings.

 2. If you host grafana under subpath make sure your grafana.ini root_path setting includes subpath

 3. If you have a local dev build make sure you build frontend using: npm run dev, npm run watch, or npm run build

 4. Sometimes restarting grafana-server can help **

Answer 1

You need to create a Gateway to allow routing between the istio-ingressgateway and your VirtualService.

Something in the lines of :

kind: Gateway
metadata:
  name: ingress
  namespace: istio-system 
spec:
  selector:
    # Make sure that the istio-ingressgateway pods have this label
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - my.domain.com      

You also need a DNS entry for your domain (my-domain.com) that points to the IP address of your istio-ingressgateway.

When your browser will hit my.domain.com, then it'll be redirected to the istio-ingressgateway. The istio-ingressgateway will inspect the Host field from the request, and redirect the request to grafana (according to VirtualService rules).

You can check kubectl get svc -n istio-system | grep istio-ingressgateway to get the public IP of your ingress gateway.

If you want to enable TLS, then you need to provision a TLS certificate for your domain (most easy with cert-manager). Then you can use https redirect in your gateway, like so :

kind: Gateway
metadata:
  name: ingress
  namespace: whatever
spec:
  selector:
    # Make sure that the istio-ingressgateway pods have this label
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - my.domain.com     
      tls:
        httpsRedirect: true
    - port: 
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - my.domain.com
      tls: 
        mode: SIMPLE
        # name of the secret containing the TLS certificate + keys. The secret must exist in the same namespace as the istio-ingressgateway (probably istio-system namespace)
        # This secret can be created by cert-manager
        # Or you can create a self-signed certificate
        # and add it to manually inside the browser trusted certificates
        credentialName: my-domain-tls

Then you VirtualService

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana
spec:
  hosts:
  - "my.domain.com"
  gateways:
  - ingress
  http:
  - match:
    - uri:
        prefix: /grafana
    route:
    - destination:
        port:
          number: 3000
        host: grafana

Answer 2

The easiest and working out of the box solution to configure that would be with a grafana host and / prefix.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: grafana-gateway
  namespace: monitoring
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http-grafana
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana-vs
  namespace: monitoring
spec:
  hosts:
  - "grafana.example.com"
  gateways:
  - grafana-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: grafana
        port:
          number: 80

---

As you mentioned in the comments, I want to use path based routing something like my.com/grafana, that's also possible to configure. You can use istio rewrite to configure that.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: grafana-gateway
  namespace: monitoring
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http-grafana
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana-vs
  namespace: monitoring
spec:
  hosts:
  - "*"
  gateways:
  - grafana-gateway
  http:
  - match:
    - uri:
        prefix: /grafana
    rewrite:
      uri: /
    route:
    - destination:
        host: grafana
        port:
          number: 80

But, according to this github issue you would have also additionally configure grafana for that. As without the proper grafana configuration that won't work correctly.

---

I found a way to configure grafana with different url with the following env variable GF_SERVER_ROOT_URL in grafana deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: grafana
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: grafana
    spec:
      containers:
      - image: docker.io/grafana/grafana:5.4.3
        name: grafana
        env:
        - name: GF_SERVER_ROOT_URL
          value: "%(protocol)s://%(domain)s/grafana/"
        resources: {}

Also there is a Virtual Service and Gateway for that deployment.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: grafana-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http-grafana
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana-vs
spec:
  hosts:
  - "*"
  gateways:
  - grafana-gateway
  http:
  - match:
    - uri:
        prefix: /grafana/
    rewrite:
      uri: /
    route:
    - destination:
        host: grafana
        port:
          number: 80