minikube ingress on macos appears to be working but never connects (times out no error)

Question

MacOS Big Sur 11.6.8 minikube version: v1.28.0

Following several tutorials on ingress and attempting to get it working locally. Everything appears to work: manual minikube service foo works, kubectl get ingress shows an IP, pinging the designated host name resolves the expected IP, etc. I went through a few different tutes with the same results.

I boiled it down to the simplest replication from the tutorial at kubernetes.io :

# kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
# kubectl expose deployment web --type=NodePort --port=8080
# kubectl get service web (ensure it's a node port)
# minikube service web --url (test url)
# kubectl apply -f ingress_hello_world.yaml
# curl localkube.com
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - host: localkube.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 8080

Manual service works:

>minikube service web --url
http://127.0.0.1:50111
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

>curl http://127.0.0.1:50111
Hello, world!
Version: 1.0.0
Hostname: web-84fb9498c7-hnphb

Ingress looks good:

>minikube addons list | grep ingress
| ingress                     | minikube | enabled ✅   | Kubernetes                     |

>kubectl get ingress
NAME              CLASS   HOSTS           ADDRESS        PORTS   AGE
example-ingress   nginx   localkube.com   192.168.49.2   80      15m

ping resolves the address mapped in /etc/hosts:

>ping localkube.com
PING localkube.com (192.168.49.2): 56 data bytes

I have looked through similar questions with no positive results. I have gone from this simple example to apache to mongo deployments via config files. Each time I can get to the app through a manual service mapping or by creating an external service (LoadBalancer / nodePort), but when I get to the Ingress part the config applies with no errors and everything appears to be working except for it actually... working.

Answer 1

Based on Veera's answer, I looked into the ingress issue with macOS and minikube tunnel. To save others the hassle, here is how I resolved the issue:

1. ingress doesn't seem to work on macOS (the different pages say "with docker" but I had the same outcome with other drivers like hyperkit.
2. the issue seems to be IP / networking related. You can not get to the minikube IP from your local workstation. If you first run minikube ssh you can ping and curl the minikube IP and the domain name you mapped to that IP in /etc/hosts. However, this does not help trying to access the service from a browser.
3. the solution is to map the domain names to 127.0.0.1 in /etc/hosts (instead of the ingress assigned IP) and use ingress components to control the domain-name -> service mappings as before...
4. then starting a tunnel with sudo minikube tunnel will keep a base tunnel open, and create tunneling for any existing or new ingress components. This combined with the ingress rules will mimic host header style connecting to any domain resolving to the local host.

Here is a full example of a working solution on mac. Dump this to a file named ingress_hello_world.yaml and follow the commented instructions to achieve a simple ingress solution that routes 2 domains to 2 different services (note this will work with pretty much any internal service, and can be a ClusterIP instead of NodePort):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
spec:
  ingressClassName: nginx
  rules:
    - host: test1.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 8080
    - host: test2.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web2
                port:
                  number: 8080

# Instructions:
# start minikube if not already
# >minikube start --vm-driver=docker
#
# enable ingress if not already
# >minikube addons enable ingress
# >minikube addons list | grep "ingress "
# | ingress                     | minikube | enabled ✅   | Kubernetes                     |
#
# >kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
# deployment.apps/web created
#
# >kubectl expose deployment web --type=NodePort --port=8080
# service/web exposed
#
# >kubectl create deployment web2 --image=gcr.io/google-samples/hello-app:2.0
# deployment.apps/web2 created
#
# >kubectl expose deployment web2 --port=8080 --type=NodePort
# service/web2 exposed
#
# >kubectl get service | grep web
# web          NodePort    10.101.19.188   <none>        8080:31631/TCP   21m
# web2         NodePort    10.102.52.139   <none>        8080:30590/TCP   40s
#
# >minikube service web --url
# http://127.0.0.1:51813
# ❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.
#
# ------ in another console ------
# >curl http://127.0.0.1:51813
#                          ^---- this must match the port from the output above
# Hello, world!
# Version: 1.0.0     <---- will show version 2.0.0 for web2
# Hostname: web-84fb9498c7-7bjtg
# --------------------------------
# ctrl+c to kill tunnel in original tab, repeat with web2 if desired
#
# ------ In another console ------
# >sudo minikube tunnel
# ✅  Tunnel successfully started
#
# (leave open, will show the following when you start an ingress component)
# Starting tunnel for service example-ingress.
# --------------------------------
#
# >kubectl apply -f ingress_hello_world.yaml
# ingress.networking.k8s.io/example-ingress created
#
# >kubectl get ingress example-ingress --watch
# NAME              CLASS   HOSTS                 ADDRESS   PORTS   AGE
# example-ingress   nginx   test1.com,test2.com             80      15s
# example-ingress   nginx   test1.com,test2.com   192.168.49.2   80      29s
#                wait for this to be populated ----^
#
# >cat /etc/hosts | grep test
# 127.0.0.1    test1.com
# 127.0.0.1    test2.com
#        ^---- set this to localhost ip
#
# >ping test1.com
# PING test1.com (127.0.0.1): 56 data bytes
#
# >curl test1.com
# Hello, world!
# Version: 1.0.0
# Hostname: web-84fb9498c7-w6bkc
#
# >curl test2.com
# Hello, world!
# Version: 2.0.0
# Hostname: web2-7df4dcf77b-66g5b

# ------- Cleanup:
# stop tunnel
#
# >kubectl delete -f ingress_hello_world.yaml
# ingress.networking.k8s.io "example-ingress" deleted
#
# >kubectl delete service web
# service "web" deleted
#
# >kubectl delete service web2
# service "web2" deleted
#
# >kubectl delete deployment web
# deployment.apps "web" deleted
#
# >kubectl delete deployment web2
# deployment.apps "web2" deleted

Answer 2

The behavior you are describing is most probably because the ingress and ingress-dns addons are currently only supported on Linux Systems when using the Docker driver, as mentioned in the Known Issues section of the minikube documentation.

Minikube supports ingress differently on the Mac and Linux.

On Linux the ingress is fully supported and therefore does not need the use of minikube tunnel.

On Mac there is an open issue due to a network issue. The documentation states that the minikube ingress addon is not supported, seems misleading if not incorrect. It's just supported differently (and not as well).

Please go through Ingress DNS and similar SO for more information.