Openvpn, forward packets very slowly

Question

I rebooted my server, and an odd issue just came out. I am running on ArchLinux, the clients are Ubuntu, Android and Mac.

The problem is that accessing the internet via the clients is slow, about 2ko/s and slowly stop.

But downloading something from the server to the client directly is made at full speed. And, obviously, internet from the server is at his full speed (40mo/s).

I don't know what happened from the reboot, but this issue is here on all clients, and is only related to the traffic that openvpn forward to internet.

EDIT: Tried with tcp, did not solve. EDIT: Tested various fragment/mtu settings, no changes.

Here are all my confs:

╭─<root@Alduin>-</etc/openvpn>-<1:45:07>-◇
╰─➤ cat Alduin.conf ccd/Thunderaan
local 212.83.129.104
port 1194
proto udp
dev tun
ca keys/ca.crt
cert keys/Alduin.crt
key keys/Alduin.key
dh keys/dh1024.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "dhcp-option DNS 10.8.0.1"
client-to-client
keepalive 5 60
ping-timer-rem
comp-lzo
persist-key
persist-tun
status openvpn-status.log
verb 3
client-config-dir ccd
topology subnet

ccd from here +++++++++++++++


ifconfig-push 10.8.0.2 255.255.255.0
push "redirect-gateway def1"

Client conf:

client
dev tun
proto udp
remote 212.83.129.104 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert name.crt
key name.key
ns-cert-type server
comp-lzo
verb 3

and some output that might help you:

╭─<cubox@Alduin>-<~>-<1:49:43>-◇
╰─➤ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether b8:ac:6f:94:e2:4e brd ff:ff:ff:ff:ff:ff
    inet 88.190.15.135/24 scope global eno1
       valid_lft forever preferred_lft forever
    inet 212.83.129.104/32 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 2001:bc8:300a:dead::b12d/64 scope global
       valid_lft forever preferred_lft forever
    inet6 2a01:e0b:1000:15:baac:6fff:fe94:e24e/64 scope global dynamic
       valid_lft 2592000sec preferred_lft 604800sec
    inet6 fe80::baac:6fff:fe94:e24e/64 scope link
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether b8:ac:6f:94:e2:4f brd ff:ff:ff:ff:ff:ff
6: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none
    inet 10.8.0.1/24 brd 10.8.0.255 scope global tun0
       valid_lft forever preferred_lft forever
╭─<cubox@Alduin>-<~>-<1:49:47>-◇
╰─➤ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         88-190-15-1.rev 0.0.0.0         UG    0      0        0 eno1
10.8.0.0        *               255.255.255.0   U     0      0        0 tun0
88.190.15.0     *               255.255.255.0   U     0      0        0 eno1
╭─<cubox@Alduin>-<~>-<1:49:51>-◇
╰─➤ route -6
Kernel IPv6 routing table
Destination                    Next Hop                   Flag Met Ref Use If
::1/128                        ::                         U    256 0     0 lo
2001:bc8:300a:dead::/64        ::                         U    256 0     0 eno1
2a01:e0b:1000:15::/64          ::                         UAe  256 0     0 eno1
fe80::/64                      ::                         U    256 0     0 eno1
::/0                           fe80::225:45ff:fef6:947f   UGDAe 1024 2     0 eno1
::/0                           ::                         !n   -1  1  1891 lo
::1/128                        ::                         Un   0   2  5227 lo
2001:bc8:300a:dead::/128       ::                         Un   0   1     0 lo
2001:bc8:300a:dead::b12d/128   ::                         Un   0   1   131 lo
2a01:e0b:1000:15::/128         ::                         Un   0   1     0 lo
2a01:e0b:1000:15:baac:6fff:fe94:e24e/128 ::                         Un   0   3 29356 lo
fe80::/128                     ::                         Un   0   1     0 lo
fe80::baac:6fff:fe94:e24e/128  ::                         Un   0   1   311 lo
ff00::/8                       ::                         U    256 0     0 eno1
::/0                           ::                         !n   -1  1  1891 lo



-A POSTROUTING -s 10.8.0.0/24 -o eno1 -j MASQUERADE # The iptables rule

The iptables rule here is the only that is active on the server.

╰─➤ tc qd
qdisc mq 0: dev eno1 root
qdisc pfifo_fast 0: dev tun0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1

EDIT: Here is a log from the Archlinux client connecting.

Oct  2 16:54:17 Groat ovpn-openvpn[9216]: OpenVPN 2.2.1 x86_64-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [eurephia] [MH] [PF_INET6] [IPv6 payload 20110424-2 (2.2RC2)] built on Feb 13 2013
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: LZO compression initialized
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: Control Channel MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: Socket Buffers: R=[212992->131072] S=[212992->131072]
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: Local Options hash (VER=V4): '41690919'
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: Expected Remote Options hash (VER=V4): '530fdded'
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: UDPv4 link local: [undef]
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: UDPv4 link remote: [AF_INET]212.83.129.104:1194
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: TLS: Initial packet from [AF_INET]212.83.129.104:1194, sid=edfcb034 3452d72c
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: VERIFY OK: depth=1, /C=FR/ST=FR/L=Paris/O=Dragonborn/CN=Dragonborn_CA/emailAddress=cubox@cubox.me
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: VERIFY OK: nsCertType=SERVER
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: VERIFY OK: depth=0, /C=FR/ST=FR/L=Paris/O=Dragonborn/CN=Dragonborn/emailAddress=cubox@cubox.me
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 1024 bit RSA
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: [Dragonborn] Peer Connection Initiated with [AF_INET]212.83.129.104:1194
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: SENT CONTROL [Dragonborn]: 'PUSH_REQUEST' (status=1)
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: PUSH: Received control message: 'PUSH_REPLY,dhcp-option DNS 10.8.0.1,route 212.83.129.0 255.255.255.0,route-gateway 10.8.0.1,topology subnet,ping 5,ping-restart 60,redirect-gateway def1,ifconfig 10.8.0.3 255.255.255.0'
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: OPTIONS IMPORT: timers and/or timeouts modified
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: OPTIONS IMPORT: --ifconfig/up options modified
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: OPTIONS IMPORT: route options modified
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: OPTIONS IMPORT: route-related options modified
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: ROUTE default_gateway=192.168.1.254
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: TUN/TAP device tun0 opened
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: TUN/TAP TX queue length set to 100
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: /sbin/ifconfig tun0 10.8.0.3 netmask 255.255.255.0 mtu 1500 broadcast 10.8.0.255
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: /sbin/route add -net 212.83.129.104 netmask 255.255.255.255 gw 192.168.1.254
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: /sbin/route add -net 0.0.0.0 netmask 128.0.0.0 gw 10.8.0.1
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: /sbin/route add -net 128.0.0.0 netmask 128.0.0.0 gw 10.8.0.1
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: /sbin/route add -net 212.83.129.0 netmask 255.255.255.0 gw 10.8.0.1
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: Initialization Sequence Completed

EDIT: Here is a tcpdump of the server downloading directly a file: http://sprunge.us/aaJX Here is the client downloading this ressource: http://sprunge.us/WUCC and here is a normal client from another openvpn (working) server: http://www4.slashusr.com/57552.tcpdump

EDIT: As asked in comments, here are raw tcpdump captures. The tun0 capture from the server failed, I don't know why. Server showing outside here, client showing tun0 here, client showing outside here and server downloading directly the file here.

EDIT: The server is running an i3, which is not used at anytime (not even during openvpn use). Same for the client, i7 totaly idle.

EDIT: The issue is still here. Please, help :(

I assume you have have looked at some captures with wireshark/tcpdump? The answer almost certainly can be found in a capture, if you capture in the right place. — Zoredache, Sep 30 '13 at 05:00
I have a tcpdump from the eno1 interface on a download from the client and one from the server (of the same file). And one from a working openvpn client too. I'll edit the question. — Cubox, Sep 30 '13 at 12:13
Can you add cpu information from the client and server while the traffic is being transferred? — Jed Daniels, Sep 30 '13 at 15:17
In your tcpdump, I don't see slow traffic (it might be too short tho). Every client gets the same ip-address 10.8.0.2? Can you omit that and rather push a route to your network 212.83.129.0? — ott--, Oct 01 '13 at 16:15
Each client have his own ccd with his own ip address. I don't understand what do you mean by a route to the network. — Cubox, Oct 01 '13 at 19:01
In the server-config: `push "route 212.83.129.0 255.255.255.0"`. Can you add a log from a client connecting? — ott--, Oct 02 '13 at 10:40
Can you upload the tcpdump files in binary tcpdump format? This will make it easier to analyze in a tool such as wireshark (which makes this sort of debugging MUCH easier). Also, can you please tcpdump both the internal and external sides of the connection (simultaneously, if possible)? I.e. if you have packets to compare from tun0 and from eno1, it should be a piece of cake to figure out what is wrong. — Jed Daniels, Oct 03 '13 at 14:49
When I'm troubleshooting vpn issues like this, I basically get 4 packet captures (after checking that CPU isn't bound up anywhere): 1) client inside, 2) client outside, 3) server inside, 4) server outside. On the client side, if you are pushing all traffic through the tunnel, then capture all the traffic. On the server side, filter to just that client's internal and external IPs, but don't filter any further. — Jed Daniels, Oct 03 '13 at 14:52
@JedDaniels Done! As you asked, I recorded everything. There might be garbage, especially on server showing eno1, which have a public ip. — Cubox, Oct 04 '13 at 01:03
Seems you have a loop in routes. What is the gateway for vpn clients: 88.190.15.1 or 212.83.129.104? Did you create separate table for 212.83.129.104 subnet routing (f.e. http://linux-ip.net/html/routing-tables.html)? Did you tried to use 88.190.15.135 address as server address? Also just in case: try to disable lso compression both server and client, in some cases it can become slow. — user1024, Dec 03 '16 at 06:30

score 1 · Answer 1 · edited Apr 13 '17 at 12:14

1

I am not sure if it is same cause but I think it worth to adjust about tun-mtu and mssfix as mentioned in openvpn-on-android-tcp-retransmissions-after-openvpn-server-reboot

EDIT: I found this one might be helpful too [RESOLVED] Inacceptable openVPN performance Changing a kernel parameter: net.inet.ip.fastforwarding = 1 (add in /etc/sysctl.conf on the your linux server)

edited Apr 13 '17 at 12:14

Community

1

answered Oct 20 '13 at 09:02

Wutiphong

109
3

Thanks for the answer. Changing the tun-mtu and mssfix option did not helped. The fastworwarding setting don't exist in Linux. Only BSD kernels. – Cubox Oct 21 '13 at 11:59

score 0 · Answer 2 · answered Oct 01 '13 at 06:09

0

Is the vpn server also the gateway server ? try removing the push-gateway, you clients need to only have an additional route.

answered Oct 01 '13 at 06:09

Alin Andrei

127
9

Where do you see a push-gateway option? – Cubox Oct 01 '13 at 16:02
At CCD options there is redirect gateway. You need yo check if there is a static route on clients to theirs real GW. – MealstroM Oct 07 '13 at 09:24
There is. Clients are able to talk to anything on the internet, they just do it slowly. – Cubox Oct 08 '13 at 22:33

score 0 · Answer 3 · answered Oct 04 '13 at 08:18

0

Your postrouting iptables rule looks weird, try this

-A POSTROUTING -s 10.8.0.0/24 -o eno1 -j MASQUERADE

instead of the SNAT, and delete one of the IPs on eno1, if you dont need them both. Two public IP addresses on what looks to be a ethernet-interface just looks weird. Why that setup?

My guess is your openvpn-server is looping and dropping packets back and forth, leading to this problem.

answered Oct 04 '13 at 08:18

AntonioP

196
6

I have the masquerade rule now, it don't resolved the issue. I have two is on my server, with one of them being the failover and the other the main one. For having a server with four ips on the eth0 interface (on another Archlinux server) for over a year, I can tell you that multiples ips are not the issue here... – Cubox Oct 04 '13 at 12:06

score 0 · Answer 4 · answered Jan 21 '14 at 00:12

0

Do you run your own dns server internally? I had issues with my network when I was running a powerdns setup for internal dns but didn't have a reverse zone configured correctly. Wireshark provided the answers in that case.

answered Jan 21 '14 at 00:12

Peter Nunn

452
1
11
25

Openvpn, forward packets very slowly

4 Answers4