0

Env info: I build my k8s cluster with virtualbox on mac.The node os is centos7.3. There are two node and one master, all of which's network is Net(can visit the public network) and Host-Only (can visit the inner network).The IP info is following:
master:
network enp0s3 :192.168.99.100/24 (Host-only network,node1 and node2 can visit this IP)
network enp0s8 :10.0.3.15/24 (Net network)
node1:
network enp0s3 :192.168.57.3/24 (Host-only network,master and node1 can visit this IP)
network enp0s8 :10.0.3.16/24(Net network)
node2:
network enp0s3 :192.168.58.2/24(Host-only network, master and node1 can visit this IP)
network enp0s8 :10.0.3.17/24(Net network)

k8s version is:
kubernetes(v1.5.2),ectd( 3.1.7),flannel(0.7.0)。

Network set on master:
etcdctl set /atomic.io/network/config '{"Network":"172.17.0.0/16"}'

flannel set on node1:
/run/flannel/subnet.env

FLANNEL_NETWORK=172.17.0.0/16
FLANNEL_SUBNET=172.17.94.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=true

/etc/sysconfig/flanneld:

FLANNEL_ETCD_ENDPOINTS="http://192.168.99.100:2379"
FLANNEL_ETCD_PREFIX="/atomic.io/network"
FLANNEL_OPTIONS="-iface=enp0s3 -public-ip=192.168.57.3 -ip-masq=true"

flannel set on node2: /run/flannel/subnet.env :

FLANNEL_NETWORK=172.17.0.0/16
FLANNEL_SUBNET=172.17.50.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=true

/etc/sysconfig/flanneld:

FLANNEL_ETCD_ENDPOINTS="http://192.168.99.100:2379"
FLANNEL_ETCD_PREFIX="/atomic.io/network"
FLANNEL_OPTIONS="-iface=enp0s3 -public-ip=192.168.58.2 -ip-masq=true"

node1's route:

flannel0:172.17.94.0/16
docker0:172.17.94.1/24

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.3.2 0.0.0.0 UG 100 0 0 enp0s8
10.0.3.0 0.0.0.0 255.255.255.0 U 100 0 0 enp0s8
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 flannel0
172.17.94.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
192.168.57.0 0.0.0.0 255.255.255.0 U 100 0 0 enp0s3

and

default via 10.0.3.2 dev enp0s8 proto static metric 100 
10.0.3.0/24 dev enp0s8 proto kernel scope link src 10.0.3.16 metric 100 
172.17.0.0/16 dev flannel0 proto kernel scope link src 172.17.94.0 
172.17.94.0/24 dev docker0 proto kernel scope link src 172.17.94.1 
192.168.57.0/24 dev enp0s3 proto kernel scope link src 192.168.57.3 metric 100 

node2's route:
flannel0: 172.17.50.0/16
docker0: 172.17.50.1/24

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.3.2 0.0.0.0 UG 0 0 0 enp0s8
10.0.3.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s8
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 enp0s8
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 flannel0
172.17.0.0 0.0.0.0 255.255.0.0 U 1 0 0 flannel0
172.17.50.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
192.168.58.0 0.0.0.0 255.255.255.0 U 100 0 0 enp0s3

and

default via 10.0.3.2 dev enp0s8 
10.0.3.0/24 dev enp0s8 proto kernel scope link src 10.0.3.17 
169.254.0.0/16 dev enp0s8 scope link metric 1003 
172.17.0.0/16 dev flannel0 
172.17.0.0/16 dev flannel0 scope link metric 1 
172.17.50.0/24 dev docker0 proto kernel scope link src 172.17.50.1 
192.168.58.0/24 dev enp0s3 proto kernel scope link src 192.168.58.2 metric 100 

Then, Ping node2's docker ip(172.17.50.1 ) on node1 is not ok, Ping node1's docker ip(172.17.94.1) is not ok either. Use tcpdump to see the network, found maybe the network is not config ok.The source ip and des ip should be 192.168.57.3(through enp0s3) but not 10.0.3.16(through enp0s8): enter image description here

I don't know why the nodes can't visit througt flannel.Hoping for help, thanks.

liyj144
  • 110
  • 7
  • Is there reason your using v1.5.2 version. latest is 1.6.4 its comes with kubeadm tool to configure the cluster. – sfgroups Jun 14 '17 at 11:50
  • flannel setup is based on etcd. is your etcd cluster working to begin with? – MrE Jun 14 '17 at 21:57
  • @sfgroups, thanks. I will try to upgrade to 1.6 and hava a look. – liyj144 Jun 15 '17 at 00:49
  • @MrE, yes the etcd cluster works well. `etcdctl ls -r | grep subnets` shows /atomic.io/network/subnets /atomic.io/network/subnets/172.17.92.0-24 /atomic.io/network/subnets/172.17.96.0-24 and `etcdctl get /atomic.io/network/subnets/172.17.92.0-24` shows ` {"PublicIP":"192.168.57.3"} ` – liyj144 Jun 15 '17 at 00:50

0 Answers0