Ingress Niginx on Multi-Node Virtualbox Driver Minikube

Question

I am following this tutorial for setting up Ingress with Ingress-Nginx on Minikube. But I can't seem to get it to work. I get a connection refused when I try to connect to port 80 on the VM IP address returned by minikube ip

My setup is this:

Minikube version: v1.25.1
VirtualBox version: 6.1
Kubernetes version: v1.22.5

The ingress-nginx namespace has the below resources:

NAME                                            READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-controller-85f4c5b458-2dhqh   1/1     Running   0          49m

NAME                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             NodePort    10.102.88.109   <none>        80:30551/TCP,443:31918/TCP   20h
service/ingress-nginx-controller-admission   ClusterIP   10.103.134.39   <none>        443/TCP                      20h

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   1/1     1            1           20h

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-85f4c5b458   1         1         1       20h

NAME                                       COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   1/1           6s         20h
job.batch/ingress-nginx-admission-patch    1/1           6s         20h

The default namespace has the below resources

NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
pod/web-79d88c97d6-rvp2r   1/1     Running   0          47m   10.244.1.4   minikube-m02   <none>           <none>

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP          20h   <none>
service/web          NodePort    10.104.20.14   <none>        8080:31613/TCP   20h   app=web

NAME                                        CLASS   HOSTS              ADDRESS     PORTS   AGE
ingress.networking.k8s.io/example-ingress   nginx   hello-world.info   localhost   80      20h

Minikube is exposing these services:

|---------------|------------------------------------|--------------|-----------------------------|
|   NAMESPACE   |                NAME                | TARGET PORT  |             URL             |
|---------------|------------------------------------|--------------|-----------------------------|
| default       | kubernetes                         | No node port |
| default       | web                                |         8080 | http://192.168.59.106:31613 |
| ingress-nginx | ingress-nginx-controller           | http/80      | http://192.168.59.106:30551 |
|               |                                    | https/443    | http://192.168.59.106:31918 |
| ingress-nginx | ingress-nginx-controller-admission | No node port |
| kube-system   | kube-dns                           | No node port |
| kube-system   | registry                           | No node port |
|---------------|------------------------------------|--------------|-----------------------------|

In step 4 of the Create an Ingress section The tutorial mentions this:

Add the following line to the bottom of the /etc/hosts file on your computer (you will need administrator access):

172.17.0.15 hello-world.info

Note: If you are running Minikube locally, use minikube ip to get the external IP. The IP address displayed within the ingress list will be the internal IP.

It's a three node cluster using VirtualBox. I've tried adding the Minikube ingress-nginx-controller service's IP (192.168.59.106, which is also the result of minikube ip) to my hosts file, but it doesn't work. And as far as I know, I can't include the service's node port 30551 in the hosts file to test that.

Some guidance on how to get this working would be much appreciated

score 2 · Answer 1 · answered Jan 24 '22 at 16:23

You are correct. You cannot include the port in the /etc/hosts file. To get there, you would need to specify the full path in your browser or some other application as following (assuming no connectivity issues):

hello-world.info:30551

I'd recommend you to tell specifically what type of issue you have. There can be multiple issues and each one will have different solution.

For example there will be a difference between the inability to access the Service and getting the 404 message.

I'm not sure if it's related but I had connectivity issues when I created a cluster in a following way:

minikube start --driver="virtualbox"
minikube node add
minikube node add

However, when I ran below command, I encountered noone:

minikube start --driver="virtualbox" --nodes=3

Assuming that you would like to expose your Nginx Ingress controller to be available on the ports 80 and 443 instead of NodePort's you can do:

Spawn your cluster
Deploy:
- Metallb.universe.tf
Configure your address pool similar to:

apiVersion: v1                                                                                                                                     
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.59.200-192.168.59.210"

Change the Service of your: ingress-nginx-controller to LoadBalancer instead of NodePort (kubectl edit svc -n ingress-nginx ingress-nginx-controller)
Check on the Service:
- kubectl get svc -n ingress-nginx ingress-nginx-controller

NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   10.106.63.253   192.168.59.201   80:30092/TCP,443:30915/TCP   23m

Put the EXTERNAL-IP of your Ingress controller to your /etc/hosts file.
Create an Ingress resource that is matching what you've inputted as a name to /etc/hosts and has some backend.

Additional resources:

Thanks @Dawid Kluk, those are very helpful suggestions. I was getting a connection refused error. I've found a solution and will add it now. Basically, if you include the --addons=ingress flag in your minikube start command, rather than running minikube enable addons ingress after creating the cluster, port 80 on the minikube node accepts connections — cjt, Jan 26 '22 at 10:29

score 1 · Accepted Answer · answered Jan 26 '22 at 10:48

When following the tutorial, I enabled the ingress addon after creating my cluster by running minikube addons enable ingress

This appeared to succeed, but when trying to connect to port 80 on the IP address returned by minikube ip (which is also the ingress-nginx-controller minikube service address), I got a connection refused. This can be validated by running:

nc -zv $(minikube ip) 80

However, when I enabled ingress at the time of initial cluster creation with this command:

minikube start --driver=virtualbox \
--kubernetes-version=v1.22.5 --nodes 3 \
--addons=ingress

and then ran nc -zv $(minikube ip) 80, the connection was accepted. I'm not sure if this is an issue with Minikube or with VirtualBox, but enabling ingress at the initial cluster creation time rather then subsequently worked for me

I was then able to update my hosts file with just the IP of the minikube node and the hello-world.info host

One thing that might also be worth noting if you create and delete your clusters a lot, I found sometimes when updating the hosts file on a Mac that old IPs were being cached. Running sudo dscacheutil -flushcache may help with this

Ingress Niginx on Multi-Node Virtualbox Driver Minikube

2 Answers2