Bare-metal k8s ingress with nginx-ingress

Question

I can't apply an ingress configuration.

I need access a jupyter-lab service by it's DNS

• http://jupyter-lab.local

It's deployed to a 3 node bare metal k8s cluster

• node1.local (master)
• node2.local (worker)
• node3.local (worker)

Flannel is installed as the Network controller

I've installed nginx ingress for bare metal like this

• kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/baremetal/deploy.yaml

When deployed the jupyter-lab pod is on node2 and the NodePort service responds correctly from http://node2.local:30004 (see below)

I'm expecting that the ingress-nginx controller will expose the ClusterIP service by its DNS name ...... thats what I need, is that wrong?

This is the CIP service, defined with symmetrical ports 8888 to be as simple as possible (is that wrong?)

---
apiVersion: v1
kind: Service
metadata:
  name: jupyter-lab-cip
  namespace: default
spec:
  type: ClusterIP
  ports:
    - port: 8888
      targetPort: 8888
  selector:
    app: jupyter-lab

• The DNS name jupyter-lab.local resolves to the ip address range of the cluster, but times out with no response. Failed to connect to jupyter-lab.local port 80: No route to host

• firewall-cmd --list-all shows that port 80 is open on each node

This is the ingress definition for http into the cluster (any node) on port 80. (is that wrong ?)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jupyter-lab-ingress
  annotations:
    # nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io: /
spec:
  rules:
  - host: jupyter-lab.local
    http:                       
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: jupyter-lab-cip
            port:
              number: 80

This the deployment

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jupyter-lab-dpt
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jupyter-lab
  template:
    metadata:
      labels:
        app: jupyter-lab
    spec:
      volumes:
        - name: jupyter-lab-home
          persistentVolumeClaim:
            claimName: jupyter-lab-pvc
      containers:
        - name: jupyter-lab
          image: docker.io/jupyter/tensorflow-notebook
          ports:
            - containerPort: 8888
          volumeMounts:
            - name: jupyter-lab-home
              mountPath: /var/jupyter-lab_home
          env:
            - name: "JUPYTER_ENABLE_LAB"
              value: "yes"

I can successfully access jupyter-lab by its NodePort http://node2:30004 with this definition:

---
apiVersion: v1
kind: Service
metadata:
  name: jupyter-lab-nodeport
  namespace: default
spec:
  type: NodePort
  ports:
    - port: 10003
      targetPort: 8888
      nodePort: 30004
  selector:
    app: jupyter-lab

How can I get ingress to my jupyter-lab at http://jupyter-lab.local ???

• the command kubectl get endpoints -n ingress-nginx ingress-nginx-controller-admission returns :

ingress-nginx-controller-admission 10.244.2.4:8443 15m

---

Am I misconfiguring ports ?

Are my "selector:appname" definitions wrong ?

Am I missing a part

How can I debug what's going on ?

---

Other details

• I was getting this error when applying an ingress kubectl apply -f default-ingress.yml

  Error from server (InternalError): error when creating "minnimal-ingress.yml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-contr
  oller-admission.ingress-nginx.svc:443/networking/v1beta1/ingresses?timeout=10s": context deadline exceeded
  

  This command kubectl delete validatingwebhookconfigurations --all-namespaces removed the validating webhook ... was that wrong to do?

• I've opened port 8443 on each node in the cluster

Answer 1

Ingress is invalid, try the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jupyter-lab-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: jupyter-lab.local
    http:                       # <- removed the -
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
#            name: jupyter-lab-cip
            name: jupyter-lab-nodeport
            port:
              number: 8888
---
apiVersion: v1
kind: Service
metadata:
  name: jupyter-lab-cip
  namespace: default
spec:
  type: ClusterIP
  ports:
    - port: 8888
      targetPort: 8888
  selector:
    app: jupyter-lab

---

If I understand correctly, you are trying to expose jupyternb through ingress nginx proxy and to make it accessible through port 80.

Run the folllowing command to check what nodeport is used by nginx ingress service:

$ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME                       TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller   NodePort   10.96.240.73   <none>        80:30816/TCP,443:31475/TCP   3h30m

In my case that is port 30816 (for http) and 31475 (for https).

Using NodePort type you can only use ports in range 30000-32767 (k8s docs: https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). You can change it using kube-apiserver flag --service-node-port-range and then set it to e.g. 80-32767 and then in your ingress-nginx-controller service set nodePort: 80

apiVersion: v1
kind: Service
metadata:
  annotations: {}
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
    nodePort: 80         # <- HERE
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
    nodePort: 443         # <- HERE
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: NodePort

Although this is genereally not advised to change service-node-port-range since you may encounter some issues if you use ports that are already open on nodes (e.g. port 10250 that is opened by kubelet on every node).

---

What might be a better solution is to use MetalLB.

---

EDIT:

How can I get ingress to my jupyter-lab at http://jupyter-lab.local ???

Assuming you don't need a failure tolerant solution, download the https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/baremetal/deploy.yaml file and change ports: section for the deployment object like following:

  ports:
    - name: http
      containerPort: 80
      hostPort: 80         # <- add this line
      protocol: TCP
    - name: https
      containerPort: 443
      hostPort: 443        # <- add this line
      protocol: TCP
    - name: webhook
      containerPort: 8443
      protocol: TCP

and apply the changes:

kubectl apply -f deploy.yaml

Now run:

$ kubectl get po -n ingress-nginx ingress-nginx-controller-<HERE PLACE YOUR HASH>  -owide
NAME                                        READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
ingress-nginx-controller-67897c9494-c7dwj   1/1     Running   0          97s   172.17.0.6   <node_name>   <none>           <none>

Notice the <node_name> in NODE column. This is a node's name where the pod got scheduled. Now take this nodes IP and add it to your /etc/hosts file.

It should work now (go to http://jupyter-lab.local to check it), but this solution is fragile and if nginx ingress controller pod gets rescheduled to other node it will stop working (and it will stay lik this until you change the ip in /etc/hosts file). It's also generally not advised to use hostPort: field unless you have a very good reason to do so, so don't abuse it.

---

If you need failure tolerant solution, use MetalLB and create a service of type LoadBalancer for nginx ingress controller.

I haven't tested it but the following should do the job, assuming that you correctly configured MetalLB:

kubectl delete svc -n ingress-nginx ingress-nginx-controller
kubectl expose deployment -n ingress-nginx ingress-nginx-controller --type LoadBalancer