0

I'd like my HAProxy Load Balancer to be able to handle at least 1M connections. I need this to deploy a stress test.

I tried setting:

maxconn 1000000

in my haproxy.cfg but HAProxy failed to start with:

[haproxy.main()] Cannot raise FD limit to 2000029, limit is 1048576.

I'm running HAProxy in a Docker Container on Ubuntu Linux 20.04 LTS that is an Amazon AWS Lightsail VPS instance.

I've looked at various questions on the internet on how to manually increase open files limit (ulimit -n) and copy pasted numerous configurations into my host. Nothing worked - I couldn't increase this limit to anything higher than 1048576.

I've found the following related posts on the subject of hard limit for the number of open files:

This question seems almost identical to mine:

Am I approaching the issue from the wrong angle? Is there another way to configure HAProxy so that it can handle a million connections (that is, without increasing ulimit -n)?

Janez Kuhar
  • 103
  • 6
  • There isn't. The title of your question says "more than 500000", which seems to be possible, but not the double. You do realize that with 1M open files, your server will be frozen? Any idea how much memory is needed for a single TCP/IP connection? – Gerard H. Pille Nov 06 '21 at 23:43
  • I was looking at [this article](https://www.freecodecamp.org/news/how-we-fine-tuned-haproxy-to-achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27/#haproxy-configuration) where they set the `maxconn` to 2,000,000. The systems they ran haproxy on had 30GB of RAM or more. – Janez Kuhar Nov 07 '21 at 06:27
  • They're not using a container on a cloud server, are they? – Gerard H. Pille Nov 07 '21 at 08:27

1 Answers1

0

Solution

The open files limit can be increased.

… the per-process hard limit is set to 1048576 by default, but it can be changed using the "fs.nr_open" sysctl. From: HAProxy Management Guide | 5. File-descriptor limitations

I've stumbled upon a useful resource on how to increase fs.nr_open. This is what I ended up doing (on my host machine):

# 0. ssh into my cloud instance

# 1. change to root
sudo su -
# 2. increase the limit
sysctl -w fs.nr_open=2010000
# 3. save changes and exit
sysctl -p
exit
# 4. now you are back in the user shell; you need to re-log
# from this shell as well for the changes to take effect
exit

Note that the changes made don't persist after a reboot. If someone knows how to do that, please let me know and I'll edit my answer.

I also needed to tell Docker that it's okay to open more than 1,048,576 files in the container. I was using a Docker Compose file to define my services. I simply added the following snippet to my docker-compose.yml:

services:
  proxy:
    image: haproxy
    # Begin snippet
    ulimits:
      nofile:
        soft: 2005000
        hard: 2005000
    # End snippet
    # ...
# ...

Last but not least, make sure that the host instance has at least 2GB of RAM or you'll run out of memory when you'll try to run the proxy. If you want to actually handle 1 million requests with your proxy, you'll require much more memory - something between 20 and 30 GB - feel free to test on your own. If you know how to calculate the limit, feel free to edit my answer or post a comment bellow!

Rationale on the number of open files

If we need 2,000,029 open files for 1 million connections, then lets:

  • set the proxy container limit slightly above that: 2,000,029 + 4971 = 2,005,000
    • set the host OS limit slightly above that: 2,005,000 + 5000 = 2,010,000
Janez Kuhar
  • 103
  • 6