Kubernetes cluster VirtualBox issues with networking (NAT and Host-only adapters)

Question

I am trying to setup a kubernetes cluster (two nodes, 1 master, 1 worker) on VirtualBox. My host computer runs Windows 10 and on the VirtualBox I have installed Ubuntu 18.10, Codename cosmic.

I have configured two adapters on each VirtualBox, one NAT and one Host-Only adapter. I did that because I need to access some internal resources using the host IP (NAT) and I also need a stable network between the host and the virtual machines (Host-only network).

I have installed Kubernetes v1.12.4 and successfully joined the worker to the master node.

NAME                STATUS   ROLES    AGE   VERSION
kubernetes-master   Ready    master   36m   v1.12.4
kubernetes-slave    Ready    <none>   25m   v1.12.4

I am using Flannel for networking.

All pods seems to be ok.

 NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
default       nginx-server-7bb6997d9c-kdcld               1/1     Running   0          27m
kube-system   coredns-576cbf47c7-btrvb                    1/1     Running   1          38m
kube-system   coredns-576cbf47c7-zfscv                    1/1     Running   1          38m
kube-system   etcd-kubernetes-master                      1/1     Running   1          38m
kube-system   kube-apiserver-kubernetes-master            1/1     Running   1          38m
kube-system   kube-controller-manager-kubernetes-master   1/1     Running   1          38m
kube-system   kube-flannel-ds-amd64-29p96                 1/1     Running   1          28m
kube-system   kube-flannel-ds-amd64-sb2fq                 1/1     Running   1          37m
kube-system   kube-proxy-59v6b                            1/1     Running   1          38m
kube-system   kube-proxy-bfd78                            1/1     Running   0          28m
kube-system   kube-scheduler-kubernetes-master            1/1     Running   1          38m

I have deployed nginx to verify that everything is working

NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   41m
nginx-http   ClusterIP   10.111.151.28   <none>        80/TCP    29m

However when I try to reach nginx I am getting a timeout. describe pod gives me the following events.

Events:
  Type     Reason                  Age                 From                       Message
  ----     ------                  ----                ----                       -------
  Normal   Scheduled               32m                 default-scheduler          Successfully assigned default/nginx-server-7bb6997d9c-kdcld to kubernetes-slave
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dbb2595628fc2579c29779e31e27e27eaeff2dbcf2bdb68467c47f22a3590bd0" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "801e0f3f8ca4a9b7cc21d87d41141485e1b1da357f2d89e1644acf0ecf634016" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "77214c757449097bfbe05b24ebb5fd3c7f1d96f7e3e9a3cd48f3b37f30224feb" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ebffdd723083d916c0910489e12368dc4069dd99c24a3a4ab1b1d4ab823866ff" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d87b93815380246a05470e597a88d50eb31c132a50e30000ab41a456d1e65107" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3ef233ef0a6c447134c7b027747a701d6576a80e76c9cc8ffd8287e8ee5f02a4" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6b621aab3c57154941b37360240228fe939b528855a5fe8cd9536df63d41ed93" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fa992bde90e0a1839180666bedaf74965fb26f3dccb33a66092836a25882ab44" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "81f74f687e17d67bd2853849f84ece33a118744278d78ac7af3bdeadff8aa9c7" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m (x2 over 32m)   kubelet, kubernetes-slave  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "29188c3e73d08e81b08b2258254dc2691fcaa514ecc96e9df86f2e61ba455b76" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          32m (x11 over 32m)  kubelet, kubernetes-slave  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling                 32m                 kubelet, kubernetes-slave  pulling image "nginx"
  Normal   Pulled                  32m                 kubelet, kubernetes-slave  Successfully pulled image "nginx"
  Normal   Created                 32m                 kubelet, kubernetes-slave  Created container

I have tried to do the same exactly installation with a bridge adapter only configured to the virtual machines and then everything works as expected.

I believe that its a configuration issue however I am unable to solve it. Can someone advise me.

How you installed flannel and kubernetes itself? If by Kubeadm, then did you pass ```--pod-network-cidr=``` option? — Anton Kostenko, Jan 22 '19 at 17:22
The command I have executed is `sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.56.120 --kubernetes-version=stable-1.12 --apiserver-cert-extra-sans=10.0.2.15` — Iraklis Kyriakidis, Jan 23 '19 at 09:13
and for flannel `kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml` — Iraklis Kyriakidis, Jan 23 '19 at 09:33

score 1 · Answer 1 · edited Jan 28 '19 at 13:28

As I have mentioned in deleted comment, I recreated this on my Ubuntu 18.04 host. Created two Ubuntu 18.10 VM, with two adapters (NAT and one Host-Only adapter). I have the same configuration as you have specified here. Everything works fine.

What I had to do was to add the second adapter manually, I did it by using netplan before running kubeadm init and kubeadm join on node.

Just in case you did not do that - add the host only adapter network to the yaml file in /etc/netplan/50-cloud-init.yaml and run sudo netplan generate and sudo netplan apply. For nginx I have used deployment from official Kubernetes documentation. Then I have exposed the service:

kubectl create service nodeport nginx --tcp=80:80 Curling my node IP address on NodePort from host machine works fine.

This was just to demonstrate what I did so it works in my environment. Judging from the described pod error it seems like there is something wrong with Flannel itself:

/run/flannel/subnet.env: no such file or directory

I checked this directory on master and it looks like this:

/run/flannel/subnet.env

FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

Check if the file is there, and if this will not help you, we can try to further troubleshoot if you provide more information. However there are too many unknowns so I had to guess in some places, my advice would be to destroy it all and try again with the information I have provided, and run the nginx with NodePort and not ClusterIP type. ClusterIP will only be reachable from inside of the cluster - for example Node.

@Iraklis Kyriakidis did this help? – aurelius Feb 04 '19 at 11:30 — aurelius, Feb 04 '19 at 11:30

score 0 · Answer 2 · answered Oct 29 '20 at 14:15

Please let me pump up this thread. Long time ago I had configurated 1 NAT for internet, 1 HOST for SSH remote and errors the same. Special when setup Rancher Longhorn.

Now, I don't build like that. First, I build the GATEWAY SERVER by using CentOS with iptable (1 NAT, 1 HOST)

Then, other VMs has just 1 interface HOST connected direct to GATEWAY SERVER

Kubernetes cluster VirtualBox issues with networking (NAT and Host-only adapters)

2 Answers2