microk8s Ingress can't access services (503)

Question

I'm running microk8s v1.18.5 from snap on Ubuntu 20.04 with addons ingress, dns, dashboard, helm3, storage.

I'm trying to externally access running services such as grafana and dashboard. I've configured proxy services and ingresses as per docs:

kind: Service
apiVersion: v1
metadata:
  name: grafana
  namespace: ingress
spec:
  type: ExternalName
  externalName: monitoring-grafana.kube-system.svc.cluster.local
  ports:
    - port: 80
---
kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: grafana-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    kubernetes.io/tls-acme: "true"
spec:
  tls:
    - hosts:
        - "grafana.example.com"
      secretName: grafana-tls
  rules:
    - host: grafana.example.com
      http:
        paths:
          - backend:
              serviceName: grafana
              servicePort: 80
            path: /
---

and

kind: Service
apiVersion: v1
metadata:
  name: dashboard
  namespace: ingress
spec:
  type: ExternalName
  externalName: kubernetes-dashboard.kube-system.svc.cluster.local
  ports:
    - port: 443
---
kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: dashboard-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    kubernetes.io/tls-acme: "true"
spec:
  tls:
    - hosts:
        - "dashboard.example.com"
      secretName: dashboard-tls
  rules:
    - host: dashboard.example.com
      http:
        paths:
          - backend:
              serviceName: dashboard
              servicePort: 443
            path: /
---

Trying to access either dashboard or grafana I get:

503 Service Temporarily Unavailable
openresty/1.15.8.1

What can I do to find the root cause?

I'm also running cert-manager and external-dns from helm3, could their config be related to the issue?

Can you post output of `kubectl get svc -A`? Are your pods running? — kool, Jul 24 '20 at 14:17
@KFC_ my output: https://pastebin.com/iDkH1CEc and thanks for taking a look. Pods are all running. I suspect ingress namespace is not what I needed to put the proxies in even though nginx-ingress-microk8s-controller is in ingress namespace. — demiters, Jul 24 '20 at 23:03

kool · Answer 1 · 2020-07-29T15:36:29.690

First you have to change your dashboard and grafana service type to NodePort for ingress to work correctly.

Besides that Kubernetes dashboard for microk8s is accessible under <master_node_ip>:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ path so you have to mention it in either your URL or in your ingress manifest. When you curl dashboard.example.com it gives you 503 Service Temporarily Unavailable error. However when you enter full path it will show the website:

curl http://dashboard.example.com:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

<!--
Copyright 2017 The Kubernetes Authors.

[...]

This is an example of ingress that will rewrite /api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy to /dashboard/

kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: grafana-ingress
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: nginx
    # Add https backend protocol support for ingress-nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Accept-Encoding "";
      sub_filter '<base href="/">' '<base href="/dashboard/">';
      sub_filter_once on;
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: dashboard.example.com
      http:
        paths:
          - path: /dashboard(/|$)(.*)
            backend:
              serviceName: kubernetes-dashboard
              servicePort: 443

microk8s Ingress can't access services (503)

1 Answers1