How to configure Nginx to proxy more than 200K MQTT connections?

Question

I need to proxy a lot of MQTT connections (over 200K) and distribute them over a number of ports at the backend server. Below is my Nginx configuration.

load_module /usr/lib/nginx/modules/ngx_stream_module.so;

user www-data;
pid /run/nginx.pid;

worker_rlimit_nofile 204800;
worker_cpu_affinity auto;
worker_processes auto;

events {
    accept_mutex on;             # Multiple workers should accept connection
    worker_connections 206200;   # Total connections with upstream and clients
    multi_accept off;            # Accept one connection in each worker
}

stream {
    upstream backend {
          server backendserver.com:1885;
          server backendserver.com:1886;
          server backendserver.com:1887;
          server backendserver.com:1888;
          server backendserver.com:1889;
          server backendserver.com:1890;
          server backendserver.com:1891;
          server backendserver.com:1892;
          server backendserver.com:1893;
          server backendserver.com:1894;
          server backendserver.com:1895;
          server backendserver.com:1896;
          server backendserver.com:1897;
          server backendserver.com:1898;
          server backendserver.com:1899;
          server backendserver.com:1900;
          server backendserver.com:1901;
          server backendserver.com:1902;
          server backendserver.com:1903;
          server backendserver.com:1904;
    }

    server {
        listen 8883;
        proxy_pass backend;
    }
}

I also have set the following system configurations:

sysctl -w fs.file-max=11000000
sysctl -w fs.nr_open=11000000
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535
sysctl -w net.ipv4.ip_local_port_range="1025 65535"
sysctl -w net.ipv4.tcp_mem="100000000 100000000 100000000"
sysctl -w net.ipv4.tcp_rmem='2048 2048 2048'
sysctl -w net.ipv4.tcp_wmem='4096 4096 4096'
sysctl -w net.core.rmem_default=4096
sysctl -w net.core.wmem_default=4608
sysctl -w net.core.rmem_max=4096
sysctl -w net.core.wmem_max=4608

Also I have set ulimit -n 999999.

But I only can create around 128K-130K connections. After that, Nginx logs erros like below:

2023/08/16 06:46:32 [error] 4172305#4172305: *206569 recv() failed (104: Unknown error) while proxying and reading from upstream, client: 72.xxx.xxx.xxx, server: 0.0.0.0:8885, upstream: "139.xxx.xxx.xxx:8889", bytes from/to client:984/5224, bytes from/to upstream:5224/984

I checked directly connecting to all of the backend ports and they work fine.

Furthermore, when I only had one entry in upstream backend block, I could only create around 64K connections. When I add the second entry, I could go upto around 128K-130K. But then adding more entries does not let me go beyond that number of connections.

I have already tried doing below:

Set worker_rlimit_nofile and worker_connections to 999999.
Add more server blocks with different ports.
10x tcp_rmem, tcp_wmem, rmem_default, wmem_default, rmem_max, wmem_max.
Setting multi_accept on;

How can I make my Nginx server to proxy more than 130K connections (at least 200K)?

How to configure Nginx to proxy more than 200K MQTT connections?

0 Answers0