Per-user routing doesn't seem to work

Question

So, I'm trying to implement per-user routing so that I'm able to route all btpd torrent traffic over a VPN. Unfortunately, btpd does not currently allow you to bind to a specific IP address. :(

I decided to try to follow this guide.

Basically, you mark any packets and then do SNAT or DNAT or MASQUERADE to the packets, then use ip rules to force a specific route table.

In the end, my setup:

(To my router)
eth1  Link encap:Ethernet  HWaddr 00:0a:cd:18:8a:ae  
      inet addr:172.29.5.10  Bcast:172.29.5.255  Mask:255.255.255.0
      inet6 addr: fe80::20a:cdff:fe18:8aae/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:2073338425 errors:0 dropped:0 overruns:0 frame:0
      TX packets:2031270514 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000 
      RX bytes:3014149031 (2.8 GiB)  TX bytes:1897259705 (1.7 GiB)
      Interrupt:17 Base address:0x6c00 

(my VPN)
tun0  Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
      inet addr:10.1.1.4  P-t-P:10.1.1.4  Mask:255.255.255.0
      UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
      RX packets:7842859 errors:0 dropped:0 overruns:0 frame:0
      TX packets:8040846 errors:0 dropped:41433 overruns:0 carrier:0
      collisions:0 txqueuelen:100 
      RX bytes:1544016294 (1.4 GiB)  TX bytes:1737654278 (1.6 GiB)

iptables -t mangle -vnL (this is where I mark the packets):

Chain PREROUTING (policy ACCEPT 42 packets, 3595 bytes) pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 42 packets, 3595 bytes) pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 36 packets, 3461 bytes) pkts bytes target     prot opt in     out     source               destination         
0     0 MARK       icmp --  *      *       0.0.0.0/0            0.0.0.0/0           owner UID match 1000 MARK set 0x2a 
0     0 MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:!53 owner UID match 1000 MARK set 0x2a 
0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           owner UID match 1000 MARK set 0x2a 

Chain POSTROUTING (policy ACCEPT 36 packets, 3461 bytes) pkts bytes target     prot opt in     out     source               destination

My iptables -t nat (Where I switch the source address):

Chain PREROUTING (policy ACCEPT 10 packets, 536 bytes) pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 10 packets, 536 bytes) pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 6 packets, 360 bytes) pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 6 packets, 360 bytes) pkts bytes target     prot opt in     out     source               destination         
0     0 MASQUERADE  all  --  *      tun0    0.0.0.0/0            0.0.0.0/0

Output of ip rule show:

0:      from all lookup local 
32761:  from all fwmark 0x2a lookup 42 
32762:  from 10.1.0.0/16 lookup 42 
32766:  from all lookup main 
32767:  from all lookup default

And finally, output of ip route show table 42:

default via 10.1.1.1 dev tun0

Now, everything seems to work fine when I do a tcpdump on my VPN interface at first. But, then it become apparent that my programs are not recieving the SYN-ACK of the threeway hand shake.

This is the output of the user with uid 1000 running curl ifconfig.me:

21:34:08.585901 IP 10.1.1.4.55707 > www1465.sakura.ne.jp.http: S 342876994:3342876994(0) win 5840 <mss 1460,sackOK,timestamp 2892988166 0,nop,wscale 6>
21:34:08.852160 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892988166>
21:34:11.587175 IP 10.1.1.4.55707 > www1465.sakura.ne.jp.http: S 342876994:3342876994(0) win 5840 <mss 1460,sackOK,timestamp 2892991168 0,nop,wscale 6>
21:34:11.848906 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892991168>
21:34:14.890209 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892991168>
21:34:17.603173 IP 10.1.1.4.55707 > www1465.sakura.ne.jp.http: S 342876994:3342876994(0) win 5840 <mss 1460,sackOK,timestamp 2892997184 0,nop,wscale 6>
21:34:17.863614 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892997184>
21:34:20.901869 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892997184>
21:34:26.962022 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892997184>
21:34:39.099655 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892997184>

As you can see, the masquerading seems to be working and I'm getting a return response. However, curl acts like it is not receiving the response. This is quickly verified with a netstat -an | grep SYN which shows curl was trying to use my internel network, not the VPN.

tcp        0      1 172.29.5.10:58000       219.94.163.75:80        SYN_SENT

I've tried doing SNAT and DNAT versus MASQUERADE to the same exact result. At this point, I'm not sure why curl is not recieving a response. Thanks.

Wouldn't it just be easier to fix `btpd` so it could bind to a specific IP address? — womble, Aug 25 '11 at 06:54
Just to be clear, this has been solved. I had disabled rp_filter on both eth1 and tun0, but I did not disable it on "all". Disabling rp_filter on all, as well as on the individual interfaces was what was required. — Nick, Aug 26 '11 at 16:59

score 2 · Accepted Answer · edited Apr 13 '17 at 12:14

2

Please verify that rp_filter is disabled. See section 10.1 of the LARTC HOWTO.. Also see this answer on a similar question

Also, if you do a ip route show table main | grep 'dev tun0' you will probably see a route like below. You need to create a route like this your table 41.

10.1.1.0/24  proto kernel  scope link dev tun0 src 10.1.1.4

edited Apr 13 '17 at 12:14

Community

1

answered Aug 25 '11 at 06:23

Zoredache

130,897
41
276
420

2

Sorry, should have specified above that reverse path filtering is disabled. I stand corrected. rp_filter was diabled on the individual interfaces, but it ALSO had to be disabled under all as well. – Nick Aug 25 '11 at 06:31

score 2 · Answer 2 · answered Jan 13 '12 at 15:31

The solution is to disable rp_filter as said earlier. But you shouldn't disable rp_filter in vain. I tried this and found out you need to switch it off only on the "per-user" interface, which was ppp0 in my case:

echo 0 | sudo tee /proc/sys/net/ipv4/conf/ppp0/rp_filter

Of course you can do it more correct by putting it to /etc/sysctl.conf.

Per-user routing doesn't seem to work

2 Answers2