minikube start fails with "libmachine: Error dialing TCP: dial tcp 10.43.239.243:49167: connect: no route to host" error inside docker in docker

Question

minikube start fails with error libmachine: Error dialing TCP: dial tcp 10.43.239.243:49167: connect: no route to host when run in the below setup:

• k8s cluster (with containerd as container runtime) with 2 pods: one with docker client container, second with docker daemon container.

dind daemon resources:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dind  
spec:
  selector: 
    matchLabels:
      app: dind
  serviceName: "dind"
  template:
    metadata:
      labels:
        app: dind
    spec:
      containers:
        - name: dind-daemon 
          image: docker:20.10.17-dind
          securityContext: 
            privileged: true
          env: 
            - name: DOCKER_TLS_CERTDIR
              value: "" 

apiVersion: v1
kind: Service
metadata:
  name: dind  
spec:
  selector:
    app: dind
  type: ClusterIP
  ports:
  - name: daemon
    protocol: TCP
    port: 2375
    targetPort: 2375

dind client resources:

apiVersion: v1
kind: Pod
metadata:
  name: "docker-client"
  labels:
    app: "docker-client"
spec:
  containers:
  - name: docker-client
    image: "docker:latest"
    env:
    - name: DOCKER_HOST
      value: "tcp://dind:2375"

• minikube start runs inside docker client container

How to debug this issue and what might be the reason for it? 10.43.239.243 is ip of ClusterIP dind service. The error happens after lines in minikube log:

I0804 09:46:35.049413     222 main.go:134] libmachine: About to run SSH command:
sudo hostname minikube && echo "minikube" | sudo tee /etc/hostname

I tried to make the same experiment when both containers run without kubernetes (using docker daemon). In that case, both were using the same docker network, daemon container started with dind network alias and minikube start succeeded.

Below are the used commands:

• docker daemon container:

docker run --name dind -d --privileged --network dind --network-alias dind -e DOCKER_TLS_CERTDIR="" docker:dind

• docker client container:

docker run --name dind-client -it  --network dind  -e DOCKER_HOST="tcp://dind:2375"docker sh
/ # wget https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
/ # mv minikube-linux-amd64 minikube
/ # chmod +x minikube 
/ # ./minikube start --force
...
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
/ # ./minikube kubectl -- run --image=hello-world
/ # ./minikube kubectl -- logs pod/hello
Hello from Docker!