0

I need to configure server to handle more than one million opened websocket connections (ideally two million).

I used configuration from this blogpost:

sysctl -w fs.file-max=12000500
sysctl -w fs.nr_open=20000500
ulimit -n 20000500
sysctl -w net.ipv4.tcp_mem='10000000 10000000 10000000'
sysctl -w net.ipv4.tcp_rmem='1024 4096 16384'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16384'
sysctl -w net.core.rmem_max=16384
sysctl -w net.core.wmem_max=16384

However, my application stops apply new connections after it reaches 469219 connections. What did I miss? I really think that something is missing in OS configuration. Our main app is written in Java (with Tomcat server) but I also got pretty similar results with NodeJS server.

We use Ubuntu with 16GB RAM.

EDIT: At the peak system uses about 12Gb of 14.7Gb.

UPD:

So finally I've got 32GB workstation. Problem solved by increasing RAM space. Currently with 18GB Java heap in use I am able to handle 567K WS connections. For higher numbers I need more clients :-)

Robbie
  • 143
  • 6
Petr Shypila
  • 143
  • 6
  • 1
    Have you looked in your dmesg.log and message.log?? you maybe hitting contrack limits. If you are it should be spamming you logs with errors about it – Nath Dec 21 '16 at 23:52
  • 1
    With that many connections the receive windows is going to mean a lot of committed memory. I don't know the details of how that is tracked, but if every one of those 469219 connections were using 16KB for receive and 16KB for send you'd already be using more than 14GB of memory. – kasperd Dec 22 '16 at 10:27
  • 3
    I'm voting to close this question as off-topic because errors and omissions in third party information should be directed to their author so that they can be corrected. – user9517 Dec 22 '16 at 10:27
  • @Hanginoninquietdesperation I do not ask to find a mistake in third party configuration. I asked if there is something what I am missing. – Petr Shypila Dec 22 '16 at 10:29
  • 1
    @Hanginoninquietdesperation I gotta disagree with that. This question contains sufficient information that it can reasonably be answered as is by anybody who knows enough about how the TCP implementation in Linux works. And citing which tutorial the settings came from does not make it a worse question. – kasperd Dec 22 '16 at 10:33
  • @kasperd well, default tcp connection size is 4KB, not 16. In this case only about 3.7GB will be used. Which is actually make sense, because exactly this amount of memory was used when I used NodeJS as a websocket server. In this case I also was not able to open higher amountof connections. Moreover, it was even less. – Petr Shypila Dec 22 '16 at 10:35
  • @Nath dmesg is empty and syslog doesn't contain any info about reaching connections limit or any other error. Today I am most likely going to receive 32GB RAM workstation to test my application. I will give results later. – Petr Shypila Dec 22 '16 at 10:46
  • I've answered your question but I really think you are being rather naive about the level of performance and capability you think you can get. – Matthew Ife Dec 22 '16 at 11:30
  • @PetrShypila I just tested a connection between two Ubuntu 14.04 systems with the default settings and they both used an initial window size of around 29KB. I am not quite sure where they got that value from, none of the mentioned sysctls seem to have a value of 29KB specified anywhere. You should try capturing some network traffic to see what window sizes are actually used by your system. – kasperd Dec 22 '16 at 11:55

1 Answers1

8

Not necessarily an answer but too big to put in a comment.

tcp_mem (since Linux 2.4)
      This  is  a vector of 3 integers: [low, pressure, high].  These bounds, measured in units of the system page size, are used by
      TCP to track its memory usage.  The defaults are calculated at boot time from the amount of available memory.  (TCP  can  only
      use  low memory for this, which is limited to around 900 megabytes on 32-bit systems.  64-bit systems do not suffer this limi-
      tation.)

      low       TCP doesnât regulate its memory allocation when the number of pages it has allocated globally is below this  number.

      pressure  When the amount of memory allocated by TCP exceeds this number of pages, TCP moderates its memory consumption.  This
                memory pressure state is exited once the number of pages allocated falls below the low mark.

      high      The maximum number of pages, globally, that TCP will allocate.  This value overrides any other limits imposed by the
                kernel.

Note the following:

These bounds, measured in units of the system page size

Setting that value to 10000000 10000000 10000000 is stating to the kernel to use 39062 MiB of memory for TCP. Nearly triple what you have.

The second problem is the 3 values for TCP rmem and wmem you set define the min, default and max. Given that your tcp_mem configuration states you never goes into 'memory saving' mode I imagine that you are actually allocating somewhere between 4-16k per socket.

So, if I was the kernel and I saw such insane settings I might not behave that predictably either.

Try reducing that value down to something you can actually use and trying again.

Finally, I will point out that you are living in a dream world if you seriously believe that:

  1. The kernel will support 2 million connections with any comfort.
  2. Node or java will support 2 million connections with any comfort.

Even under best circumstances (using an epoll set) 2 million entries in an epoll set is expensive. Thats never going to happen with a worker or prefork model.

You need to be spreading this load out more evenly. You probably need another 10 nodes at least to get anything worthy of what a user would call a service.

Matthew Ife
  • 23,357
  • 3
  • 55
  • 72
  • I agree with one exception, 500k * 16k would fit on 8gb Ram, so the rmem/wmem settings are not too problematic. – eckes Aug 07 '19 at 19:38