4

On my VPS (Ubuntu 14.04 - 64-bit), which has OpenVPN 2.3.2-server installed, I'm getting disappointing performance. The VPN-server uses the AES-128-CBC cipher for encryption, and I have no firewall installed.

The problem is that the download/upload-speed is rather low. While doing some speed-tests, the average is around 45 Mbps. The actual speed will drop down to 20 Mbps, and go as high as 50 Mbps. While doing these tests, the CPU and server-load seem rather low.

I've tried changing the tun-mtu value on both the server and client, using different ciphers (blowfish etc.), upping the buffersize etc. My download-speed without the VPN-connection is around 180 - 190 Mbps (my ISP promises 200 Mbps), my upload is 10 Mbps. As you can see the difference is rather drastic and I'd like to up the download-speed as much as possible.

Using programs such as haveged from generating available entropy, no performance seems to be gained. My available entropy is always around 700 - 800 bits 3000 with haveged active, which is plenty enough?

The server itself has a 1 Gbit connection. A download test to the server confirms this speed (around 800+ Mbps). I've also tested downloading a random 1 GB file from the server to one of the VPN-clients without using the VPN-connection, and this gave me a download-speed of 180 Mbps (my maximum) on average.

I couldn't find much information on the internet about slow/low download-speeds with an OpenVPN-server that were helping. The thing that concerns me is that the CPU and server-load are very low (CPU usage is about 16 % on only one core (OpenVPN does not support multi/hyper-threading?) while doing speed tests).

If you'd like more information or details about my set-up or configuration, please ask.

EDIT 2015-02-28:

I'm once again trying to use an OpenVPN-server with only three (3) clients. Again, I'm facing the problem where the download speed does not max (not even half) my internet connection speed on the clients.

I've tested installing the OpenVPN-server on the host machine itself, and that gave me quite good results! But, I need to use the VPN server on a VPS (virtual machine) and not on a host machine. While downloading a 10 GB file, I can go as high as 10 MB/s (800 Mb/s), and averages at about 7,5 MB/s (60 Mb/s). My internet connection where the clients are connecting from is a 200 Mb/s connection. When doing various speed test without a VPN connection, I can reach an average of 180 Mb/s. While connecting through the VPN server (on the VPS!), the speeds at not even half the speed I should get. When connecting through the VPN server (on the HOST!), I can reach up to 120 Mb/s.

Both servers use about the same CPU usage while performing the tests and both have aesni support on the CPU. I've done some speed tests with openssl, but those seem very good and max. out CPU cores. I've also run watch -n 1 cat /proc/sys/kernel/random/entropy_avail while performing these tests, but the available entropy seems to not change that much while performing a download test. On both the host and the VPS it's around 850 - 900.

I'm out of ideas as to why the VPS server(s) differ that much in speed from the host. Could it be the Qemu (qemu-kvm) virtualised network interface?

Qlii256
  • 151
  • 2
  • 8
  • Still no solution. Speed even further decreased to 10 and less Mbps. – Qlii256 Nov 07 '15 at 18:24
  • I've tried multiple entropy-generating-programs (such as haveged, mentioned below), but none seem to really help. The available entropy stays at around 800 bits. I've tried disabling fragment and mssfix (--fragment=0 --mssfix=0) and upping the mtu (--tun-mtu=2000), but also this does not affect my performance. I've looked more deeply into the systems' load while performing speed tests, but it doesn't seem to go that high. Using a higher cpu-clock or more cores doesn't really affect performance. The speeds sometimes go up to 50 - 55 Mbps, but then go back down to 20 or even 10 or lower Mbps. – Qlii256 Nov 08 '15 at 11:18
  • As you can read in my question now, the available entropy is around 3100 bits while having haveged active. It's a lot higher then the 800 bits while it's not active, but still does not seem to improve my speeds. I also think that the 800 bits is plenty enough, it never runs out of low while performing speed tests. – Qlii256 Nov 08 '15 at 12:44
  • How is the CPU usage on both OpenVPN endpoints (client and server) during the speed test? How much is the latency between endpoints? Test the latency when the link is saturated and when it is not saturated. 4 test in total. – Mircea Vutcovici Nov 08 '15 at 13:08
  • Thank you for your comment. The CPU usage is rather low ( I see OpenVPN service on server with 12.5 - max. 15 % CPU uage while performing a test and very low (as my clients' CPU is much better then the servers' on my client. The latency from client to server is around 30 ms while on VPN-connection. I do not really understand what you mean with 'saturated'. Could this be encrypted and non-encrypted? Also worth mentioning is I'm using LZO-compression, turning it off does not really affect performance. – Qlii256 Nov 08 '15 at 13:13
  • 1
    It is interesting to see how the latency is degrading when you do a network test over VPN. It is important to see the if the degradation is only inside the OpenVPN link or also outside. If the degradation is only in OpenVPN, then you should focus on OpenVPN performance. – Mircea Vutcovici Nov 08 '15 at 13:16
  • Well, I did some test running download tests on testmy.net. I did 5 with VPN and 5 without VPN. Without VPN I always got around 180 - 185 Mbps download with a really stable connection (speed stays the same for the whole download). With the VPS however, the speed did not stay stable at all. It would go from 50 Mbps to 20 to 25 and so on. Most of the time it's around 25 Mbps. The servers' CPU load is very low however (talking about 15 % usage on one of two cores (2 GHz)). Also worth mentioning, speed test was a 100 MB download. – Qlii256 Nov 08 '15 at 14:52
  • I've done a lot more tests. While installing OpenVPN-server on a host (hardware server), the speeds are amazing! But, whenever I install it onto the same host (or another) but inside virtual machine, the speeds drop. I've been looking at the CPU flags the vm's have, and made sure they match the hosts' flags, but it doesn't help speed-wise. – Qlii256 Nov 10 '15 at 23:34
  • Performing an OpenSSL-test gives me this: "aes-256-cbc 59376.04k 61532.07k 61917.01k 69324.46k 61560.15k". So, it appears that when encrypting AES (with AES-NI CPU-flag enabled), it does about 60 MB/s, that equals to 480 Mbps, which is plenty enough for my max. ISP speed of 200 Mbps. I know encryption isn't all, but this test just shows that encryption, nor the CPU's speed is bottlenecking my VPN performance. – Qlii256 Nov 10 '15 at 23:36
  • Encryption is done in CPU or, sometimes, offloaded in specialized co-processors (ASICs). It could be that only one CPU is saturated and that could be your bottleneck, even if the overall CPU average is low. When looking for performance problems you have to monitor both the guest and the host. – Mircea Vutcovici Nov 11 '15 at 00:50
  • I've take a look at my clients and all of them use very powerful CPU's which do not really care about the OpenVPN process. Also worth mentioning is that I've tested all my clients on a 'real' hardware server which was capable of doing 180 Mpbs over VPN for all clients. The VPS itself is just missing something that the host has, and that is causing the bad performance. The CPU usage on that VM is just very low, but the usage is only on one core (I've noticed this). However, 15 % CPU usage is very low. After doing a OpenSSL write test you can see it's capable of doing 60 MB/s = 480 Mbps. – Qlii256 Nov 11 '15 at 11:15
  • In my experience, OpenVPN speeds on Gigabit channels are always limited by CPU. It's single thread, rather slow process, it can't exceed 300-400 megabit even with compression/encryption disabled. – Sergei Jun 28 '17 at 13:08

2 Answers2

4

Whenever I see someone on a VPS, and they mention encryption or SSL performance problems, the first thing that comes to mind is they're running the random pool dry.

Try installing "haveged" and see if that fixes the problem. If it does, carefully read the documentation and the caveats of using a pseduo random number generator and it's security implications.

EDIT

Why install "haveged"? Any encryption operation depends heavily on high quality random numbers. Therefore, your throughput will eventually be limited by the rate at which you can generate them. On a VPS, you may not have all the entropy sources available that you would on bare metal. This can exhaust the pool (try cat /proc/sys/kernel/random/entropy_avail), causing your program to block while more randomness is stirred into the pool. Haveged uses the HAVEGE algorithm which, depending on your use case, may be less secure. It'll be up to you to decide it's worth the risks that may apply to your situation. :) Haveged is a daemon that runs in the background to keep the kernel entropy pool full and stirred. No change to your software should be required other than maybe a reboot of your VPS (depends on your setup).

Jonathan S. Fisher
  • 434
  • 4
  • 19
  • Thanks for sharing that information! I'll try "haveged", although I know nothing about it. Can you further explain as to why this could possibly help my situation? Do you mean I should use "haveged" instead of OpenSSL for encryption? – Qlii256 Nov 02 '15 at 21:26
  • No, haveged just keeps the entropy pool full that openssl sips from. They do different things to you can't replace one with the other. – Jonathan S. Fisher Nov 02 '15 at 21:36
  • Thank you for the information! I've read your edit and I'll try this tomorrow. The security is not that big of a problem, as long as my data is still encrypted over the internet in a a way that it is not just plain text and not that easy to crack. One thing that I still don't fully understand is why my CPU usage is very low, considering that the pool can not keep up with the requests? – Qlii256 Nov 02 '15 at 21:39
  • Because the CPU doesn't generate entropy technically ;) It can only sample entropy, not create it (This is classical physics, unless you're a quantum physicist). – Jonathan S. Fisher Nov 02 '15 at 21:43
  • Ah, that explains it! I'll update once I've tried "haveged". Let's hope it gives me a (huge) boost :). – Qlii256 Nov 02 '15 at 22:00
  • 2
    +1 for giving me unexpected knowledge expansion. – ErikE Nov 03 '15 at 00:57
  • Also a +1 from me (my reputation was to low at first). – Qlii256 Nov 03 '15 at 05:51
  • To be clear on this, haveged is less secure but should increase performance. OpenVPN should be working fine (better) with "real" hardware instead of a VPS. – Qlii256 Nov 03 '15 at 08:28
  • I've run a speed test and executed "cat /proc/sys/kernel/random/entropy_avail" to get the available entropies. It was at 751 just before starting the test, and going up to 883 while doing the test. I have no idea if this is normal, but to me the available entropies are not causing my bad performance, as the entropies even go up while doing the test? – Qlii256 Nov 03 '15 at 18:11
  • It's also worth mentioning I'm running munin-node on that server, and the graph for available entropy shows me an average of 818. – Qlii256 Nov 03 '15 at 18:26
  • After giving haveged a try, it seems to not really help with my problem. I can see no difference in speeds (both upload and download). I want to thank you for the help, but sadly enough it's not really working. Maybe you can think of something else that might be the problem? Also worth mentioning is that I've already tested VPN-server set-up on 3 different VPS's, and all of them gave me the same bad performance. All the servers were on completely different hardware and 2 of them in a different datacenter. So, I still think this should be something configuration-related or a bug. – Qlii256 Nov 04 '15 at 18:44
  • Because it doesn't have any sense. After successful TCP/UDP connection there is established secured channel using asymmetrical SSL certificates/keys. Using that channel server and client negotiate connection symmetrical encryption key and digest algorithm. For the time of the session it doesn't change (it's renegotiated every hour - default setting), that's why generating that 128 bit of random data (default setting) doesn't have any influence on connection speed. Period. – Michal Sokolowski Nov 11 '15 at 07:36
0

OpenVPN over UDP

If you are running OpenVPN over UDP, you might get a better experience by setting fixed buffer values. Try these lines into your client configuration file (*.ovpn or *.conf).

sndbuf 393216
rcvbuf 393216
push "sndbuf 393216"
push "rcvbuf 393216"

https://winaero.com/blog/speed-up-openvpn-and-get-faster-speed-over-its-channel/

tom
  • 1