kubernetes nginx ingress controller return 404

Question

Following this guide, I created an ingress controller on my local kubernetes server, the only difference is that it is created as a NodePort.

I have done some test deployments, with respective services and everything works, here the file

Deploy1:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: helloworld1
spec:
  selector:
    matchLabels:
      app: helloworld1
  replicas: 1
  template:
    metadata:
      labels:
        app: helloworld1
    spec:
      containers:
      - name: hello
        image: gcr.io/google-samples/hello-app:1.0
        ports:
        - containerPort: 8080

Deploy2:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: helloworld2
spec:
  selector:
    matchLabels:
      app: helloworld2
  replicas: 1
  template:
    metadata:
      labels:
        app: helloworld2
    spec:
      containers:
      - name: hello
        image: gcr.io/google-samples/hello-app:2.0
        ports:
        - containerPort: 8080

Deploy3:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: geojson-example
spec:
  selector:
    matchLabels:
      app: geojson-example
  replicas: 1
  template:
    metadata:
      labels:
        app: geojson-example
    spec:
      containers:
        - name: geojson-container
          image: "nmex87/geojsonexample:latest"
          ports:
            - containerPort: 8080

Service1:

apiVersion: v1
kind: Service
metadata:
  name: helloworld1
spec:
#  type: NodePort
  ports:
  - port: 8080
  selector:
    app: helloworld1

Service2:

apiVersion: v1
kind: Service
metadata:
  name: helloworld2
spec:
#  type: NodePort
  ports:
  - port: 8080
  selector:
    app: helloworld2

Service3:

apiVersion: v1
kind: Service
metadata:
  name: geojson-example
spec:
  ports:
    - port: 8080
  selector:
    app: geojson-example

This is the ingress controller:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/default-backend: geojson-example
spec:
  rules:
    - http:
        paths:
          - path: /geo
            pathType: Prefix
            backend:
              service:
                name: geojson-example
                port:
                  number: 8080
          - path: /test1
            pathType: Prefix
            backend:
              service:
                name: helloworld1
                port:
                  number: 8080
          - path: /test2
            pathType: Prefix
            backend:
              service:
                name: helloworld2
                port:
                  number: 8080

When I do a GET on myServer:myPort/test1 or /test2 everything works, on /geo i get the following answer

{
    "timestamp": "2021-03-09T17:02:36.606+00:00",
    "status": 404,
    "error": "Not Found",
    "message": "",
    "path": "/geo"
}

Why?? if I create a pod, and from inside the pod, i do a curl on geojson-example it works, but from the external, i obtain a 404 (i think by nginx ingress controller)

This is the log of nginx pod:

x.x.x.x - - [09/Mar/2021:17:02:21 +0000] "GET /test1 HTTP/1.1" 200 68 "-" "PostmanRuntime/7.26.8" 234 0.006 [default-helloworld1-8080] [] 192.168.168.92:8080 68 0.008 200 

x.x.x.x - - [09/Mar/2021:17:02:36 +0000] "GET /geo HTTP/1.1" 404 116 "-" "PostmanRuntime/7.26.8" 232 0.013 [default-geojson-example-8080] [] 192.168.168.109:8080 116 0.012 404 

What can I do?

Answer 1

As far the doc: This annotation is of the form nginx.ingress.kubernetes.io/default-backend: <svc name> to specify a custom default backend. This <svc name> is a reference to a service inside of the same namespace in which you are applying this annotation. This annotation overrides the global default backend.

This service will be handle the response when the service in the Ingress rule does not have active endpoints.

You cannot use same service as default backend and also for a path. When you do this the path /geo became invalid. As we know default backend serves only the inactive endpoints. Now If you tell that you want geojson-example as default backend(for inactive endpoints) again in the paths if you tell that use geojson-example for a valid path /geo then it became invalid as you are creating a deadlock type situation here.

You actually do not need to give this nginx.ingress.kubernetes.io/default-backend annotation.

Your ingress should be like below without the default annotation, or you can use the annotation but in that case you need to remove geojson-example from using for any valid path in the paths, or need to use another service for the path /geo. Options that you can use are given below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - http:
        paths:
          - path: /geo
            pathType: Prefix
            backend:
              service:
                name: geojson-example
                port:
                  number: 8080
          - path: /test1
            pathType: Prefix
            backend:
              service:
                name: helloworld1
                port:
                  number: 8080
          - path: /test2
            pathType: Prefix
            backend:
              service:
                name: helloworld2
                port:
                  number: 8080

Or:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/default-backend: geojson-example
spec:
  rules:
    - http:
        paths:
          - path: /geo
            pathType: Prefix
            backend:
              service:
                name: <any_other_service>    # here use another service except `geojson-example`
                port:
                  number: 8080
          - path: /test1
            pathType: Prefix
            backend:
              service:
                name: helloworld1
                port:
                  number: 8080
          - path: /test2
            pathType: Prefix
            backend:
              service:
                name: helloworld2
                port:
                  number: 8080

Or:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/default-backend: geojson-example
spec:
  rules:
    - http:
        paths:
          - path: /test1
            pathType: Prefix
            backend:
              service:
                name: helloworld1
                port:
                  number: 8080
          - path: /test2
            pathType: Prefix
            backend:
              service:
                name: helloworld2
                port:
                  number: 8080

Answer 2

This is for your default backend. You set the geojson-example service as a default backend.

The default backend is a service which handles all URL paths and hosts the nginx controller doesn't understand (i.e., all the requests that are not mapped with an Ingress).

Basically a default backend exposes two URLs:

/healthz that returns 200
/ that returns 404

So , if you want geojson-example service as a default backend then you don't need /geo path specification. Then your manifest file will be:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/default-backend: geojson-example
spec:
  rules:
    - http:
        paths:
          - path: /test1
            pathType: Prefix
            backend:
              service:
                name: helloworld1
                port:
                  number: 8080
          - path: /test2
            pathType: Prefix
            backend:
              service:
                name: helloworld2
                port:
                  number: 8080

Or if you want geojson-example as a ingress valid path then you have to remove default backend annotation. Then your manifest file will be:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - http:
        paths:
          - path: /geo
            pathType: Prefix
            backend:
              service:
                name: geojson-example
                port:
                  number: 8080
          - path: /test1
            pathType: Prefix
            backend:
              service:
                name: helloworld1
                port:
                  number: 8080
          - path: /test2
            pathType: Prefix
            backend:
              service:
                name: helloworld2
                port:
                  number: 8080