0

I'm trying to expose my application in K8s.

I have set up an Ingress Controller which gives the following properties:

kubectl get svc,pods --namespace ingress
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP           PORT(S)                      AGE
service/ingress-nginx-nginx-ingress   LoadBalancer   10.254.234.220   111.111.111.111       80:32097/TCP,443:32755/TCP   16d

NAME                                               READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-nginx-ingress-68848c49f8-54lx4   1/1     Running   0          12d
pod/ingress-nginx-nginx-ingress-68848c49f8-5dx97   1/1     Running   0          12d
pod/ingress-nginx-nginx-ingress-68848c49f8-89grn   1/1     Running   0          12d

So when i go to https://111.111.111.111 (not the real address) in my browser, I get:

enter image description here

This is the ingress for my app in values.yaml`

ingress:
  name: externalIngress
  enabled: true
  type: LoadBalancer
  ingressClassName: "nginx"
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  hosts:
    - host:
      paths:
      - path: "/"
        backend:
          serviceName: my-app
          servicePort: 80

using the template:

{{- if .Values.ingress.enabled -}}
{{- $fullName := include "my-app.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
{{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
  name: {{ $fullName }}
  labels:
    {{- include "my-app.labels" . | nindent 4 }}
  {{- with .Values.ingress.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}
spec:
  {{- if .Values.ingress.tls }}
  tls:
    {{- range .Values.ingress.tls }}
    - hosts:
        {{- range .hosts }}
        - {{ . | quote }}
        {{- end }}
      secretName: {{ .secretName }}
    {{- end }}
  {{- end }}
  rules:
    {{- range .Values.ingress.hosts }}
    - host: {{ .host | quote }}
      http:
        paths:
          {{- range .paths }}
          - path: {{ .path }}
            backend:
              serviceName: {{ $fullName }}
              servicePort: {{ $svcPort }}
          {{- end }}
    {{- end }}
  {{- end }}

I have a service set up for the ingress:

kubectl get service --namespace=ingress -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      meta.helm.sh/release-name: ingress-nginx
      meta.helm.sh/release-namespace: ingress
    creationTimestamp: "2022-04-29T14:51:36Z"
    labels:
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: ingress-nginx-nginx-ingress
      helm.sh/chart: nginx-ingress-0.10.4
    name: ingress-nginx-nginx-ingress
    namespace: ingress
    resourceVersion: "4711523"
    selfLink: /api/v1/namespaces/ingress/services/ingress-nginx-nginx-ingress
    uid: a8183382-151f-499b-b06f-0a189e302226
  spec:
    clusterIP: 10.254.234.220
    externalTrafficPolicy: Local
    healthCheckNodePort: 31343
    ports:
    - name: http
      nodePort: 32097
      port: 80
      protocol: TCP
      targetPort: 80
    - name: https
      nodePort: 32755
      port: 443
      protocol: TCP
      targetPort: 443
    selector:
      app: ingress-nginx-nginx-ingress
    sessionAffinity: None
    type: LoadBalancer
  status:
    loadBalancer:
      ingress:
      - ip: 111.111.111.111
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

I have a service set up for my app

kubectl get service --namespace=my-app -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      meta.helm.sh/release-name: my-app
      meta.helm.sh/release-namespace: my-app
    creationTimestamp: "2022-05-12T12:11:47Z"
    labels:
      app.kubernetes.io/instance: my-app
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: my-app
      app.kubernetes.io/version: 1.16.0
      helm.sh/chart: my-app-0.1.0
    name: my-app
    namespace: my-app
    resourceVersion: "7395487"
    selfLink: /api/v1/namespaces/my-app/services/my-app
    uid: d72661e1-be92-42f5-a030-65bdf4da06c8
  spec:
    clusterIP: 10.254.153.184
    ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    selector:
      app.kubernetes.io/instance: my-app
      app.kubernetes.io/name: my-app
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

And the following ingress:

kubectl get ingress --namespace=my-app -o yaml
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      meta.helm.sh/release-name: my-app
      meta.helm.sh/release-namespace: my-app
      nginx.ingress.kubernetes.io/rewrite-target: /$2
    creationTimestamp: "2022-05-16T11:38:13Z"
    generation: 3
    labels:
      app.kubernetes.io/instance: my-app
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: my-app
      app.kubernetes.io/version: 1.16.0
      helm.sh/chart: my-app-0.1.0
    name: my-app
    namespace: my-app
    resourceVersion: "8501216"
    selfLink: /apis/extensions/v1beta1/namespaces/my-app/ingresses/my-app
    uid: 27fa844e-7672-47ff-94b2-b9c18492cb5d
  spec:
    rules:
    - http:
        paths:
        - backend:
            serviceName: my-app
            servicePort: 80
          path: /
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

A clusterRole for the ingress with the following permissions: enter image description here

Based on this, I'd expect to be able to go to http://111.111.111.111/ and be routed to my-app but I just get the 404.

The ingress logs give:

 controller.go:3050[] Using the DEPRECATED annotatio │
│ n 'kubernetes.io/ingress.class'. The 'ingressClassName' field will be ignored.

and

event.go:285[] Event(v1.ObjectReference{Kind:"Ingre │
│ ss", Namespace:"my-app", Name:"my-app", UID:"27fa844e-7672-47ff-94b2-b9c18492cb5d", APIVersion:"networking.k8s.io/v1beta1", ResourceVersio │
│ n:"8499566", FieldPath:""}): type: 'Warning' reason: 'Rejected' my-app/my-app was rejected: with error: spec.rules[0].host: Required value

What am I doing wrong!?

WhatTheWhat
  • 197
  • 3
  • 13

1 Answers1

0

Its usually easier to share the yaml AFTER the templating :)

In order for this to work you need-

( in the ingress namespace)

  1. A way to get traffic from the outside world to your cluster, such as an aws load balancer - this one is infra specific
  2. svc to map traffic to the ingress controller (could be of type LoadBalancer in supported cloud deployments)
  3. nginx-ingress-controller running
  4. a kubernetes cluster role + binding that gives the ingress controller permission to see ingresses and services in every namespace

(in your app namespace)

  1. A pod with your app running in it
  2. A svc that matches your pod
  3. An ingress that matches your svc

Since you didn't mention a service, my guess is thats your issue- trying to send traffic directly to the pod instead of through a service. If my guess is wrong, the logs from the nginx ingress controller should be more clear. You probably don't need to run 3 of those especially to start btw ;)

Paul Becotte
  • 9,767
  • 3
  • 34
  • 42
  • As you can probably tell, i'm new to K8s! I have most of those these things set up, I'll update my question with more cluster info – WhatTheWhat May 16 '22 at 13:07
  • Ah ok, so the ingress logs gives: `Warning Rejected 26m (x2 over 61m) nginx-ingress-controller spec.rules[0].host: Required value`. But according the docs host is optional? This is also a confusing message, has the ingress resource been rejected or is it just a warning? – WhatTheWhat May 16 '22 at 16:27
  • I think your issue is going to be that your cluster role doesn't match your ingress- different api group. Your ingress seems to be the older beta version- updating to the current version may just make things work – Paul Becotte May 16 '22 at 16:28
  • Are you referring to `- apiVersion: extensions/v1beta1` ? This seems to be populated by the template values `{{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}} apiVersion: networking.k8s.io/v1beta1 {{- else -}} apiVersion: extensions/v1beta1` – WhatTheWhat May 16 '22 at 16:34
  • looks like the helm chart you are using may have a bug in it. If your k8s version is > 1.19 (and it should be if you're starting today!) you should be using `networking.k8s.io/v1` there. – Paul Becotte May 16 '22 at 16:36
  • I see, the k8s version is 1.16 as that's the only version supported by our infrastructure. The ingress controller is nginx-stable/nginx-ingress v0.10.4. – WhatTheWhat May 16 '22 at 16:42
  • Wow- 1.16 has been EOL for like 18 months. If my guess is right, then you could instead update the clusterrole to use the older ingress api versions. – Paul Becotte May 16 '22 at 17:12
  • so the versioning culprit turned out to be a copy/paste error, basically using a more up to date `ingress.yaml`. I regenerated the Helm chart and now have the correct ingress file. I had a few yaml errors in the `values.yaml` which i've solved. However, this has made zero difference, i still get the 404 with the ingress logs giving little help! – WhatTheWhat May 17 '22 at 13:58
  • I don’t understand how I can deploy an app using v1 but the ingress as v1beta1. I do think all these problems are down to using an older version so I’ll try upgrading/ provisioning a more up to date cluster and report back. – WhatTheWhat May 17 '22 at 17:33
  • So, you will need the kind/version of your cluster role to match the kind/version of the ingress. If you are in doubt, you could grant the ingress controller permissions to everything temporarily. If its working correctly, change your ingress (add a path, host, delete/recreate, whatever). the ingress controller should log that it is updating its config. If not, you have a permission problem. If SO, then you can exec into the container and look at the generated nginx config files and figure out why its not routing you appropriately. – Paul Becotte May 18 '22 at 03:35