Access Prometheus and Grafana via Ingress in AKS

Question

I have an AKS Cluster deployed and I have deployed Prometheus and Grafana in monitoring namespace. I have also created a Ingress rule to access both the application in default namespace. To access both the services I have created services in default namespace as External name.

kind: Service
apiVersion: v1
metadata:
  name: prometheus-server-alb
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
    - protocol: TCP
      port: 9090
      targetPort: 9090
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '3000'
spec:
  selector: 
    app: grafana  
  ports:
    - port: 3000
      targetPort: 3000
---
kind: Service
apiVersion: v1
metadata:
  name: grafana-ext
  namespace: default
spec:
  type: ExternalName
  externalName: grafana.monitoring.svc.cluster.local
  ports:
  - port: 3000
    targetPort: 3000
---
kind: Service
apiVersion: v1
metadata:
  name: prometheus-ext
  namespace: default
spec:
  type: ExternalName
  externalName: prometheus-server-alb.monitoring.svc.cluster.local
  ports:
  - port: 9090
    targetPort: 9090
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sitecore-ingress
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: grafana.test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana-ext
            port: 
              number: 3000
  - host: prometheus.test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-ext
            port: 
              number: 9090
  tls:
  - secretName: monitoring-tls
    hosts:
    - prometheus.test.com
    - grafana.test.com

I am able to access the services locally inside default namespace and also provided all the required thing link certificate, etc. Still I am not able to access it. It is giving me "502-bad gateway error".

Can anyone help me? Thank you in advance.

Answer 1

• I would suggest you to please add the below annotation in your AKS deployment yaml file to prevent using private IP for the application gateway: -

 kubernetes.io/ingress.class: azure/application-gateway
 appgw.ingress.kubernetes.io/use-private-ip: "false"

The above annotation along with the one that you have used to allow the SSL redirection for the application gateway while accessing the Graffana and Prometheus monitoring namespace while accessing the AKS cluster through the application gateway itself will surely help you get past the ‘Bad gateway’ error.

• Also, other than the above, I will request you to please check whether readiness or liveness probe is defined for the monitoring namespaces or not. As per what I can get to know, it is not enabled, thus I would request you to please enable them and then check the status. I would also suggest you to please check the default health probe configured whether it is able to reach the monitoring namespaces or not. The error that you are facing regarding the bad gateway might also be due to the health probe not able to reach the backend resources. As when you provision an application gateway, the health probe is automatically by default configured through the ‘HTTPGet’ method.

For more details regarding the above, I would request you to please refer to the following documentation links: -

https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502