containers with ipv6 addresses can't connect to outside in k8s/calico environment

Question

I am trying to test ipv6 connectivity in k8s environment, and installed calico network plugin; the issue is that the container can't ping to the ipv6 gateway or other addresses of the cluster nodes, the version of k8s and calico is v1.18.2 and calico v1.12(also tried v1.13); the configurations as followings:

centos7, kernel is 4.4(upgraded)
opened ipv6 forwarding
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1

calico config:

[root@k8s-master-01 ~]# calicoctl get ipp -owide
NAME                  CIDR            NAT    IPIPMODE   VXLANMODE   DISABLED   SELECTOR   
default-ipv4-ippool   10.244.0.0/16   true   Never      Never       false      all()      
default-ipv6-ippool   fc00:f00::/24   true   Never      Never       false      all()      

within the pod, can see ipv6 address is allocated
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1440
        inet 10.244.36.196  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::a8c6:c1ff:fe61:258c  prefixlen 64  scopeid 0x20<link>
        inet6 fc00:fd8:4bce:9a48:4ab7:a333:5ec8:c684  prefixlen 128  scopeid 0x0<global>
        ether aa:c6:c1:61:25:8c  txqueuelen 0  (Ethernet)
        RX packets 23026  bytes 3522721 (3.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24249  bytes 3598501 (3.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@k8s-worker-01 ~]# ip -6 route show
fc00:fd8:4bce:9a48:4ab7:a333:5ec8:c684 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::ecee:eeff:feee:eeee dev eth0 metric 1024 pref medium

actually, I captured the messages with tcpdump from the host, and can see some icmp requests came in to the like cali66e9f9aafee interface, but looks no furthur processing, I checked ip6tables and saw that no any packages came into the right CHAIN of masqurade

[root@k8s-worker-01 ~]# ip6tables -t nat -vnL
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    1    80 cali-PREROUTING  all      *      *       ::/0                 ::/0                 /* cali:6gwbT8clXdHdC1b1 */

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 791 packets, 63280 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  796 63680 cali-OUTPUT  all      *      *       ::/0                 ::/0                 /* cali:tVnHkvAo15HuiPy0 */

Chain POSTROUTING (policy ACCEPT 791 packets, 63280 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  796 63680 cali-POSTROUTING  all      *      *       ::/0                 ::/0                 /* cali:O3lYWMrLQYEMJtB5 */

Chain cali-OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  796 63680 cali-fip-dnat  all      *      *       ::/0                 ::/0                 /* cali:GBTAv2p5CwevEyJm */

Chain cali-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  796 63680 cali-fip-snat  all      *      *       ::/0                 ::/0                 /* cali:Z-c7XtVd2Bq7s_hA */
  796 63680 cali-nat-outgoing  all      *      *       ::/0                 ::/0                 /* cali:nYKhEzDlr11Jccal */

Chain cali-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    1    80 cali-fip-dnat  all      *      *       ::/0                 ::/0                 /* cali:r6XmIziWUJsdOK6Z */

Chain cali-fip-dnat (2 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain cali-fip-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain cali-nat-outgoing (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all      *      *       ::/0                 ::/0                 /* cali:Ir_z2t1P6-CxTDof */ match-set cali60masq-ipam-pools src ! match-set cali60all-ipam-pools dst

tried lots of times, but failed, did i missed something?

regards

How did you bootstrapped your cluster (kubeadm)? How did you activated ipv6 on it? — Mark Watney, May 13 '20 at 10:25
yes, kubeadm; and is there any dedicate methods to open ipv6 feature on centos ? I just add ipv6 address and gateway in its network interfaces, and did some simple connectivity test. — techer, May 17 '20 at 02:48
here's some key output: [root@k8s-worker-01 ~]# cat /etc/modprobe.d/disable_ipv6.conf options ipv6 disable=0 [root@k8s-worker-01 ~]# cat /etc/sysconfig/network # Created by anaconda NETWORKING_IPV6=yes [root@k8s-worker-01 ~]# sysctl -a |grep net.ipv6.conf.all.disable_ipv6 sysctl: reading key "net.ipv6.conf.all.stable_secret" net.ipv6.conf.all.disable_ipv6 = 0 [root@k8s-worker-01 ~]# sysctl -a |grep net.ipv6.conf.default.disable_ipv6 net.ipv6.conf.default.disable_ipv6 = 0 — techer, May 17 '20 at 03:08
calico bgp peers status: IPv6 BGP status +------------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +------------------+-------------------+-------+----------+-------------+ | fd00:0:3:86::102 | node-to-node mesh | up | 03:00:35 | Established | | fd00:0:3:86::103 | node-to-node mesh | up | 03:00:32 | Established | +------------------+-------------------+-------+----------+-------------+ — techer, May 17 '20 at 03:08
I have exactly the same issue. Maybe setting `CALICO_IPV6POOL_NAT_OUTGOING=true` on the calico-node container as found [here](https://github.com/projectcalico/calico/issues/2954) works for you ... for me it did not. — Kimses, May 26 '20 at 15:30
I finally got my setup working by adding a NAT rule with ip6tables manually like this: `ip6tables -t nat -A POSTROUTING -o sixxs -s fd00::/64 -j MASQUERADE` — Kimses, May 26 '20 at 16:10

score 1 · Answer 1 · answered May 28 '20 at 15:11

I had exactily the same issue with a similar CentOS7 setup.

Besides following the instruction on the calico website and securing that all nodes had ipv6 forwarding enabled the solution was setting the environment variable CALICO_IPV6POOL_NAT_OUTGOING to true for the install-cni in the initContainers section and for the calico-node in the containers section.

In my case I also had to set the IP_AUTODETECTION_METHOD to my actual interface with the public v6 IP address.

I also explicitly added --proxy-mode=iptables to the kube-proxy parameters (which I'm not sure if it is default).

I hope this helps.

hi, thanks for your comments, i found that the root cause is because calico delete the route to container automatically right after about 15 second when the route was created, like below: — techer, Jun 20 '20 at 14:16

score 0 · Answer 2 · answered May 18 '20 at 07:16

Enabling ipv6 on your cluster isn't as simple as you did. Just configuring ipv6 in your network isn't going to work with Kubernetes.

First and most important topic in this mater is that IPv4/IPv6 dual-stack is an alpha feature. As any alpha feature it may not work as expected.

Please go through this document to understand better the requisites to make it work in your cluster (you have to add a feature-gate).

There is also some work being done to make it possible to bootstrap a Kubernetes cluster with Dual Stack using Kubeadm, but it's not usable yet and there is no ETA for it.

There are some examples of IPv6 and dual-stack setups with other networking plugins in this repository.

This project serves two primary purposes: (i) study and validate ipv6 support in kubernetes and associated plugins (ii) provide a dev environment for implementing and testing additional functionality (e.g.dual-stack)

score 0 · Answer 3 · answered Jun 20 '20 at 14:18

thanks for your comments, i found that the root cause is because calico delete the route to container automatically right after about 15 second when the route was created, like below: [2020-06-20T22:12:21.292676] ff00::/8 dev caliad9673f27e9 table local metric 256 pref medium [2020-06-20T22:12:21.292723] fe80::/64 dev caliad9673f27e9 proto kernel metric 256 pref medium [2020-06-20T22:12:21.292736] 10.244.36.212 dev caliad9673f27e9 scope link [2020-06-20T22:12:21.292746] fc00:f00:0:24fe:200:8fa7:f4c7:af14 dev caliad9673f27e9 metric 1024 pref medium [2020-06-20T22:12:23.173297] local fe80::ecee:eeff:feee:eeee dev lo table local proto unspec metric 0 pref medium [2020-06-20T22:12:23.173376] local fe80:: dev lo table local proto unspec metric 0 pref medium [2020-06-20T22:12:31.734619] Deleted fc00:f00:0:24fe:200:8fa7:f4c7:af14 dev caliad9673f27e9 metric 1024 pref medium

and there's an issue report in github, they found it in calico 3.9 version, and i tried this in 3.13.4, the same result.. https://github.com/projectcalico/calico/issues/2876

containers with ipv6 addresses can't connect to outside in k8s/calico environment

3 Answers3