I need to run IPsec over a Geneve tunnel (or GRETAP). I can get the IPsec tunnel to work using public IPs as endpoints (so it's just IPsec) but not using private IPs on the Geneve link (so it's IPsec/Geneve.)
The Geneve tunnel is up and works:
[root@dd1215950359 /]# ip -d link show dev tun_gw2
6: tun_gw2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether ca:9e:0e:f0:6b:9c brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65485
geneve id 102 remote 192.168.192.2 ttl 254 dstport 6081 noudpcsum udp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
[root@dd1215950359 /]# ifconfig tun_gw2
tun_gw2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 192.168.12.2 netmask 255.255.255.254 broadcast 255.255.255.255
ether ca:9e:0e:f0:6b:9c txqueuelen 1000 (Ethernet)
RX packets 396 bytes 405632 (396.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 796 bytes 464646 (453.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
The Geneve neighbor's IP is 192.168.12.3 and I can ping it.
I create the IPsec tunnel using this script. If I use the same script but using the two GW's public IPs, it works. But when I use the Geneve private IPs, the traffic bypasses the tunnel.
GW=1 # or 2, for the other side
GW1_PUBIP=192.168.160.5
GW2_PUBIP=192.168.192.2
PRIVNET=192.168.12 # address prefix on the Geneve link
SPI=0x1234
MODE="mode tunnel"
PROTO="proto esp spi $SPI"
AUTHKEY=0x0123456789ABCDEF0123456789ABCDEF
ENCKEY=0xFEDCBA9876543210FEDCBA9876543210
if [[ $GW == 1 ]]; then
LOC_PUB_IP=$GW1_PUBIP REM_PUB_IP=$GW2_PUBIP LOC_PRI_IP=$PRIVNET.2 REM_PRI_IP=$PRIVNET.3
else
LOC_PUB_IP=$GW2_PUBIP REM_PUB_IP=$GW1_PUBIP LOC_PRI_IP=$PRIVNET.3 REM_PRI_IP=$PRIVNET.2
fi
PUBIP=$(ifconfig eth0 | grep inet | tr -s ' ' | cut -d' ' -f3)
if [[ $PUBIP == $LOC_PUB_IP ]]; then # trigger guard
ip xfrm state flush; ip xfrm policy flush
ip xfrm state add src $LOC_PRI_IP dst $REM_PRI_IP $PROTO $MODE auth sha256 $AUTHKEY enc aes $ENCKEY
ip xfrm state add src $REM_PRI_IP dst $LOC_PRI_IP $PROTO $MODE auth sha256 $AUTHKEY enc aes $ENCKEY
ip xfrm policy add dst 100.64/16 dir out tmpl src $LOC_PRI_IP dst $REM_PRI_IP $PROTO $MODE
ip xfrm policy add src 100.64/16 dir in tmpl src $REM_PRI_IP dst $LOC_PRI_IP $PROTO $MODE
ip xfrm policy add src 100.64/16 dir fwd tmpl src $REM_PRI_IP dst $LOC_PRI_IP $PROTO $MODE
fi