increase nginx active connection more than 65535

Question

currently I'm doing setup for nginx to accept C1M, I do in my Virtual BOX with Centos 7.0 and RAM 10G.

Here my sysctl.conf configuration.

fs.file-max                    = 9999999
fs.nr_open                     = 9999999

net.core.netdev_max_backlog    = 4096
net.core.rmem_max              = 16777216
net.core.somaxconn             = 3240000
net.core.wmem_max              = 16777216
net.core.netdev_max_backlog    = 16384

net.ipv4.ip_forward            = 0
net.ipv4.ip_local_port_range   = 1      65535
net.ipv4.tcp_rmem              = 8192    873800  8388608
net.ipv4.tcp_wmem              = 4096    655360  8388608
net.ipv4.tcp_mem               = 8388608 8388608 8388608
net.ipv4.tcp_fin_timeout       = 7
net.ipv4.tcp_keepalive_time    = 30
net.ipv4.tcp_max_syn_backlog   = 3240000
net.ipv4.tcp_max_tw_buckets    = 6000000
net.ipv4.tcp_no_metrics_save   = 1
net.ipv4.tcp_syn_retries       = 2
net.ipv4.tcp_tw_recycle        = 1
net.ipv4.tcp_tw_reuse          = 1
net.ipv4.tcp_slow_start_after_idle = 0

vm.min_free_kbytes             = 65536
vm.overcommit_memory           = 2
net.nf_conntrack_max           = 237680
net.netfilter.nf_conntrack_max = 237680

net.ipv4.ip_conntrack_max               = 131072
net.ipv4.netfilter.ip_conntrack_max     = 1048576

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

In nginx.conf, I set worker_connection to 1000000, worker_process 4, worker_rlimit_nofile 1000000

I set my openfile with ulimit -n 2000000, but I only get active connection about 65K, like this one.

Active connections: 65318 
server accepts handled requests
 768234 768234 1549023 
Reading: 0 Writing: 61245 Waiting: 4073 

I try with wrk -t3 -c67000 -d40s http://my-ip:80

Is there any miss configuration that I make? Thank you for your reply.

Bayu Permadi,

Xavier Lucas · Answer 1 · 2014-10-23T08:23:37.600

1

Are you running your benchmark tool from multiple machines or from a single one ? It seems you are running from a single one and so the local port range is the culprit.

edited Oct 23 '14 at 08:23

answered Oct 23 '14 at 08:17

Xavier Lucas

13,095
2
44
50

yes, I test with single machine before. I'm sorry for late comments. Thanks to your answer. I've try before with another tools that almost close with client behavior, to get real action of the user. – Bayu Permadi Sep 08 '15 at 02:11

score 0 · Accepted Answer · answered Jan 28 '15 at 12:54

Number of active connections is limited by number of socket's available.

socket is clien_ip + client_port + server_port + server_ip. Maximum port number is about 65k, so the only way to increase number of connections - use additional IP.

For example use virtual interfaces. Or Add virtual adapter to virtualbox machine.

Anyway, for good performance testing you should use distributed clients(load generators), because tests with single or several clients often becomes performance test of client, not server =)

increase nginx active connection more than 65535

2 Answers2