4

So I have a Kubernetes cluster, and I am using Flannel for an overlay network. It has been working fine (for almost a year actually) then I modified a service to have 2 ports and all of a sudden I get this about a completely different service, one that was working previously and I did not edit:

<Timestamp> <host> flanneld[873]: I0407 18:36:51.705743 00873 vxlan.go:345] L3 miss: <Service's IP>
<Timestamp> <host> flanneld[873]: I0407 18:36:51.705865 00873 vxlan.go:349] Route for <Service's IP> not found

Is there a common cause to this? I am using Kubernetes 1.0.X and Flannel 0.5.5 and I should mention only one node is having this issue, the rest of the nodes are fine. The bad node's kube-proxy is also saying it can't find the service's endpoint.

Christian Grabowski
  • 2,782
  • 3
  • 32
  • 57
  • BTW for those who think this belongs on serverfault, I thought so too, but they don't even have these tags there. – Christian Grabowski Apr 07 '16 at 18:47
  • Are you actually having problems connecting with the service or its endpoints, or just asking about log lines in flannel? Service VIPs are totally virtual and intercepted by kubeproyx, and converted to endpoint ips. As long as flannel can route those endpoint ips you *should* be fine (but there might certainly be something deeper going on). – Prashanth B Apr 07 '16 at 23:03
  • I'm asking all the above. I couldn't ping the service's ip, and those log lines repeating were the only abnormal thing. – Christian Grabowski Apr 07 '16 at 23:07
  • If you do a `cat /run/flannel/subnet.env` and then `ps aux | grep docker` do the IPs in both of those match up? – Charlino Apr 08 '16 at 00:37
  • I should mention this is also an open issue on flannel https://github.com/coreos/flannel/issues/427 – Christian Grabowski Aug 08 '17 at 16:08

1 Answers1

4

Sometime flannel will change it's subnet configuration... you can tell this if the IP and MTU from cat /run/flannel/subnet.env doesn't match ps aux | grep docker (or cat /etc/default/docker)... in which case you will need to reconfigure docker to use the new flannel config.

First you have to delete the docker network interface

sudo ip link set dev docker0 down
sudo brctl delbr docker0

Next you have to reconfigure docker to use the new flannel config.
Note: sometimes this step has to be done manually (i.e. read the contents of /run/flannel/subnet.env and then alter /etc/default/docker)

source /run/flannel/subnet.env

echo DOCKER_OPTS=\"-H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}\" > /etc/default/docker

Finally, restart docker

sudo service docker restart
Charlino
  • 15,802
  • 3
  • 58
  • 74