server to handle 70 requests a second each with a response time less than 50 milli seconds

Question

I receive a large number of http GET requests from limited number of IPs (< 4) from nearby servers. Task is to maintain a response time <= 50 milli-seconds for every request.

I have enabled TCP connection reuse by setting tcp_tw_reuse to 1. ip_local_port_range is set to 1024 to 65535.

tcp_fin_timeout is set to 60 (default).

In my webserver conf file (nginx), I have set keepalive_timeout to 5 (Is this anywhere related to tcp's TIME_WAIT?).

Right now, I am receiving 5 requests a second and response time is around 200 ms.

I need help to make some significant improvements to my response time (local computation time is negligible).

Answer 1

I am going to go out and guess that these are static files and you are not passing them through a cgi.

From my experience in profiling, and googling profiling, Its all about finding the bottleneck, or optimizing the areas which take the most time, not spending all your effort to speed up the process which takes 5% of your time.

I'd like to know more about your setup. What is the response time for one file? What is the return trip time, for a ping? How big are the files?

for example if a ping takes 150ms, your problem is your network, not your nginx conf. If files are in the megabytes, its not nginx.

If the response time differs between 1 - 30 requests per second, I would assume that something more intense than finer nginx tweaks.

Can you shed any more light on the situation?

-- update -- I did a benchmark on my out of the box nginx server getting a typical index.php page.

When benchmarked from inside the server:

roderick@anon-webserver:~$ ab -r -n 1000 -c 100 http://anon.com/index.php
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking anon.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx/0.8.54
Server Hostname:        anon.com
Server Port:            80

Document Path:          /index.php
Document Length:        185 bytes

Concurrency Level:      100
Time taken for tests:   0.923 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Non-2xx responses:      1000
Total transferred:      380000 bytes
HTML transferred:       185000 bytes
Requests per second:    1083.19 [#/sec] (mean)
Time per request:       92.320 [ms] (mean)
Time per request:       0.923 [ms] (mean, across all concurrent requests)
Transfer rate:          401.96 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    4   1.6      4       9
Processing:     1   43 147.6      4     833
Waiting:        1   41 144.4      3     833
Total:          4   47 148.4      8     842

Percentage of the requests served within a certain time (ms)
  50%      8
  66%      8
  75%      9
  80%      9
  90%     13
  95%    443
  98%    653
  99%    654
 100%    842 (longest request)

When benchmarked from my home desktop:

roderick@Rod-Dev:~$ ab -r -n 1000 -c 100 http://anon.com/index.php
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking anon.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx/0.8.54
Server Hostname:        anon.com
Server Port:            80

Document Path:          /index.php
Document Length:        185 bytes

Concurrency Level:      100
Time taken for tests:   6.391 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Non-2xx responses:      1000
Total transferred:      380000 bytes
HTML transferred:       185000 bytes
Requests per second:    156.48 [#/sec] (mean)
Time per request:       639.063 [ms] (mean)
Time per request:       6.391 [ms] (mean, across all concurrent requests)
Transfer rate:          58.07 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       40  260 606.9    137    3175
Processing:    81  214 221.7    140    3028
Waiting:       81  214 221.6    140    3028
Total:        120  474 688.5    277    6171

Percentage of the requests served within a certain time (ms)
  50%    277
  66%    308
  75%    316
  80%    322
  90%    753
  95%    867
  98%   3327
  99%   3729
 100%   6171 (longest request)

My OS is linux, my cpu is 3 years old (it was a $500 server).

I have done absolutley nothing to the config file.

What does this tell me? nginx is not the problem.

Either your server's network blows or AWS is limiting your CPU. I would probably guess both.

If the fix is that important, I would get a dedicated server. But thats only as far as my knowledge goes.

Answer 2

nginx's keepalive_timeout controls the HTTP protocol's ability to re-use existing connections and has nothing to do with the TCP TIME_WAIT state. (It is also unrelated to the TCP keep-alive probes; those are sent after roughly two hours idle time and thereby useless for nearly everything.)

HTTP1.1 clients and servers (and some combinations of HTTP1.0 clients and servers) will re-use existing connections to ask for new data, which will save the time required for the TCP three-way handshake. You might wish to try increasing this timeout value if your client and server can make use of it.

The TCP TIME_WAIT state is there to make sure both peers know a dead connection is dead -- if one side re-uses the ports when re-connecting and the other side missed the FIN packet, it might think the packets from the new connection are actually for the old connection. Oops. The TIME_WAIT state prevents this. There is generally no need to fiddle with this number; consider your clients, connecting 70 times per second. With 63k ports to choose from, that's roughly 500 seconds between port re-use: 63k ports / 70 cps == 1000 seconds, random choice says perhaps half that. TIME_WAIT is close to two minutes, and this is seven or eight minutes. When you get around 100 connections per second from a peer, I'd start to worry more about TIME_WAIT.

Instead, the problem I think you've run into is Nagle's algorithm, used to prevent the Internet from being over-run by a pile of silly-small packets. Nagle's algorithm causes TCP systems to wait a little while when sending a small amount of data in the hopes that more data will arrive while waiting. This is excellent once things get going, but can cause unacceptable delays when initiating a connection. The usual mechanism to counter Nagle is to set the TCP_NODELAY socket option. (Well, fiddling with the application's patterns of send(2) and recv(2) calls is better, but not always an option. Hence, this band-aid.) See tcp(7), setsockopt(2) manpages for details. Since Nagle's algorithm interacts poorly with the TCP delayed ACK algorithm, you can also ask TCP to send immediate ACK packets, rather than the usual small delay on ACK packets: that's the TCP_QUICKACK socket option.

The tcp(7) manpage also provides a slight hint on a tcp_low_latency tunable that might do what you need. Try turning it on. (I do not know the consequences of this decision, but my guess would be influencing Nagle and Delayed ACK site-wide rather than per-socket options.)

Answer 3

Because there is so few client hosts, maybe closed TCP connections in TIME_WAIT state causes the slowness by reserving available port numbers.

Maybe you should try:

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle

Note about the option:

Enable fast recycling of TIME_WAIT sockets. Enabling this option is not recommended since this causes problems when working with NAT (Network Address Translation).