Kubernetes Ingress Controller returning 503 Service Unavailable

Question

So, I have an ingress controller routing traffic to three different services, but only one is working, all others are returning 503. INGRESS YAML

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: dev
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  rules:
  - host: localhost
    http:
      paths:
      - path: /equip
        backend:
            serviceName: web-equip-svc-2
            servicePort: 18001
      - path: /hello
        backend:
            serviceName: hello-service
            servicePort: 80
      - path: /equip-ws
        backend:
            serviceName: web-equip-svc-2
            servicePort: 18000

WORKING SVC YAML

apiVersion: v1
kind: Service
metadata:
  name: hello-service
  namespace: linkerd-test
  labels:
    app: hello
spec:
  type: ClusterIP
  selector:
    app: hello
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

NOT WORKING SVC YAML

---
apiVersion: v1
kind: Service
metadata:
  name: web-equip-svc-2
  namespace: dev
  labels:
    app: equipment-service
spec:
  type: ClusterIP
  selector:
    app: equipment-service
  ports:
  - name: "http"
    port: 18001
    targetPort: 8080
    protocol: TCP
  - name: "websocket"
    port: 18000
    targetPort: 8080
    protocol: TCP

So, I've already tried to change annotations from ingress, change svc from clusterIP to loadBalancer... and nothing worked, any help would be welcomed

Answer 1

You should keep your services as ClusterIP if you can. The point of the Ingress Controller is to have one centralised ingress into your cluster.

First thing to try

Test your services independently first. (The two that are not working). Exec into another pod that is running, and do:

curl http://web-equip-svc-2:18001 and see if you get a response back going directly to the service rather than via your ingress. If that works fine, then you know its a problem with your ingress rule configuration and/or controller.

If it doesn't work, then you know its just your actual container/pod that is running those two services, and you can focus there and fix the problem there first.

Second thing to try

Simplify your ingress rule. Remove the path for /equip-ws as a start, and have just your paths for /hello and for /equip.

      - path: /equip-ws
        backend:
            serviceName: web-equip-svc-2
            servicePort: 18000

Then test http://localhost/hello and http://localhost/equip again with the simplified ingress rule changed.

If this works then you know that your two equip paths in your ingress rule are causing issues and you can fix the conflict/issue there.

Answer 2

For me I had another ingress for the same host in another namespace (I forgot about it)

kubectl get ingress --all-namespaces | grep <HOST>

helped me find it.

Answer 3

Finally found the solution, As @Shogan suggested I split the rules on different hosts, but still no success, so instead of using Paths, I added prefixes to the host and it worked.

rules:
  - host: hello.localhost
    http:
      paths:
      - path: /hello
        backend:
          serviceName: hello-service
          servicePort: 280
  - host: equip.localhost
    http:
      paths:
      - backend:
          serviceName: web-equip-svc
          servicePort: 18001
  - host: ws.equip.localhost
    http:
      paths:
      - backend:
          serviceName: web-equip-svc
          servicePort: 18000

Answer 4

Additionally to @Shogan's answer, another thing to verify is that the port specified by the ingress matches the port specified in the service configuration.

Not applying here, @Thiago's config, but in my case the HTTP 503 was returned because of service specifying a port named http:

apiVersion: v1
kind: Service
metadata:
  ....
spec:
  selector:
    ....
  ports:
    - name: http  # here is the port name to refer inside the ingress

but the ingress specifying a port named httpS (this was a typo; I had no port named https into my service):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  ....
spec:
  ingressClassName: "nginx"
  rules:
  {{- range .Values.ingress.hosts }}
    - host: "localhost"
      http:
        paths:
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: myapp-http
                port:
                  name: https    # here was the typo

This was breaking the routing from the ingress toward the correct port of the service.
Hence the 503 HTTP status (Service Unavailable).

Although, IMHO, an *HTTP 502 (Bad Gateway)* would have been more clear on what's the cause of the issue, since the service actually is available but just not reachable.

Answer 5

make sue you service and ingress is deployed on same namspace