1

What I want to implentent is something like below.

                 192.168.1.1                                 192.168.1.2
+--------+      +----------+         +-----------+           +---------+       +---------+
|  APP0  +<---->+   tap0   +<------->+   myapp   +<--------->+   tap1  +<----->+   APP1  |
+--------+      +----------+         +-----------+           +---------+       +---------+
  1. Launch myapp, which will create two tap device, tap0 and tap1. myapp will forward packet between this two tap devices.
  2. APP0 and APP1 will communicate with standard tcp socket API.
  3. All those app and tap device is in one linux system (Centos7.5).

But after create two tap devices, ping -I tap0 192.168.1.2 not success.

First I find there is ARP problem, and I resolve them by set accept_local and rp_filter file.

echo 1 > /proc/sys/net/ipv4/conf/tap0/accept_local
echo 1 > /proc/sys/net/ipv4/conf/tap1/accept_local
echo 2 > /proc/sys/net/ipv4/conf/tap0/rp_filter
echo 2 > /proc/sys/net/ipv4/conf/tap1/rp_filter;

With tcpdump tool, I found ICMP request reached tap1, but tap1's ICMP reply went to lo device. Linux will route local traffic all through lo deivce.

How can I make local traffic skip lo device and went to the tap devices?

I did some search and find some solutions which need to config NAT policy. I think they are complicate.

randomeval
  • 111
  • 2
  • Have a look at my answer to a related question: https://serverfault.com/questions/994721/ping-does-not-work-on-tap-interfaces-with-bridge/996479#996479 . If you don't want to handle a full tcp/ip stack in your applications, use network namespaces bridges and veth interfaces instead of tap interfaces. – A.B Sep 18 '20 at 18:29
  • @A.B I can't use network namespace. If use veth pairs without namespace, it will have same problem as tap interfaces. – randomeval Sep 19 '20 at 10:06
  • Ok, so are your applications handling their tcp/ip stack? (ie: extracting IP packet from frame then payload from IP packet, sending ARP requests to know what's the MAC destination to send frames etc.) – A.B Sep 19 '20 at 11:36

1 Answers1

0

I have found an answer for this kind of problem.

The core problem is how to bypass local routing table. I found some useful info from this link.

I will show how to do it by veth interface.

  1. create veth pair interface: veth0, veth1. And set IP for them.
ip link add veth0 type veth peer name veth1
ip link set dev veth0 up
ip link set dev veth1 up
ip addr add 10.8.1.3 dev veth0
ip addr add 10.8.1.4 dev veth1
  1. Use iptables set mark for packets out from veth0 and veth1
iptables -t mangle -s 10.8.1.3/32 -A OUTPUT -j MARK --set-mark 1
iptables -t mangle -s 10.8.1.4/32 -A OUTPUT -j MARK --set-mark 1
  1. Change the priority of local rule. You better add those command when system up. The command first remove 'local' rule, this will cause your connection to linux server closed.
ip rule del from all pref 0 lookup local
ip rule add from all pref 100 lookup local
  1. Add route to table 100. This two rules just forward packets to other interface.
ip route add 10.8.1.4/32 via 10.8.1.3 dev veth0 table 100
ip route add 10.8.1.3/32 via 10.8.1.4 dev veth1 table 100
  1. Add a new rule, which have highest priority. All packets which mark 1 will first trigger this rule and lookup route in table 100.
ip rule add fwmark 1 pref 10 lookup 100
  1. You also need set accept_local and rp_filter to make ARP work properly.
echo 1 > /proc/sys/net/ipv4/conf/veth0/accept_local
echo 1 > /proc/sys/net/ipv4/conf/veth1/accept_local
echo 2 > /proc/sys/net/ipv4/conf/veth0/rp_filter
echo 2 > /proc/sys/net/ipv4/conf/veth1/rp_filter;
  1. Now you can ping each IP address, the ICMP packets will not go to lo interface.
randomeval
  • 111
  • 2