0

We have a setup that looks like this:

nginx->haproxy->app servers

We are terminating SSL with nginx and it sits in front of everything. During our peak load times, we are experiencing about a 2x performance hit. Requests that would normally take 400 ms are taking 800ms. It's taking longer for the entire Internet.

The problem is, I have absolutely no sign of any slowdowns in my logs and graphs. New Relic shows all the app servers are responding correctly with no change in speed. Nginx and haproxy show nothing in their logs about requests slowing down, but we are slowing down. Despite nginx showing that a particular request I tracked is taking 17ms through the entire stack, it took 1.5 seconds to curl it during peak load last week.

So, that leaves me with two options: 1) Network issues - I have more than enough pipe left according to graphs from the router. I'm only using 400 Mbps out of the 1 Gbps port and there are no errors in ifconfig or on the switch or routers. However, SoftLayer manages this gear, so I can't verify this personally. It could be on our side because of the kernel as well I suppose, so I'm posting my sysctl values below:

2) nginx is holding up the request and either not logging it or I'm not logging the right thing. Is it possible that requests are being queued up because workers are busier and they're not getting acted on as quickly? If this is in fact happening, what can I log in nginx other than $request_time, since that is showing no slowdown at all. And, if this is possible that requests are actually taking longer than $request_time is indicating, how do I go about tweaking the config to speed things up?

Sysctl

net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_synack_retries = 2
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fin_timeout = 3
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 16777216 16777216 16777216
net.ipv4.tcp_wmem = 16777216 16777216 16777216
net.ipv4.tcp_max_tw_buckets = 16777216
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_syn_backlog = 262144
net.core.somaxconn = 262144
net.core.netdev_max_backlog = 15000
net.core.netdev_budget = 8196
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.ip_nonlocal_bind = 1

Applicable nginx configuration

user www-data;
worker_processes 20;
worker_rlimit_nofile 500000;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
  use epoll;
  multi_accept off;
  accept_mutex off;
  worker_connections 65536;
}
Will
  • 826
  • 2
  • 9
  • 19
  • Worker processes should be <= to the number of real CPUs you have (not hyperthreading). Any more and you are adding unnecessary overhead due to context switching. It is plausible when you are busy you are delegating work to processes that are stuck in the runqueue. It may be possible to visualise this problem more acutely by checking the hosts load during peak times (which would be more or equal to num_workers-num_non_ht_cpus). Oh and change your default tcp default values. I seriously doubt you have 1TB of memory to support all those connections you could accept in nginx. – Matthew Ife Sep 28 '13 at 08:42

2 Answers2

2

You can add queue time to your newrelic graphs:

in nginx configuration at your SSL terminator add to server block:

    set $msecstart "${msec}000";
    if ($msecstart ~ "^(.*)\.(.*)") {set $msecout "t=$1$2";}
    proxy_set_header X-Request-Start $msecout;

So X-Request-Start header will contain time in microseconds and when this request will get to newrelic agent, it will update the graphs. Make sure the time well synced both at the balancer and backend servers.

ps. 000 trick is needed because $msec in nginx is in MILLIseconds and newrelic agent expects data in MICROseconds.

Andrei Mikhaltsov
  • 3,027
  • 1
  • 23
  • 31
  • Thanks. I stumbled across that earlier and will be adding it to my New Relic for sure, but it doesn't quite answer my question. I'm trying to find out if the total amount of time that it is taking through nginx and all the way down and back is reflected in $request_time, or if there's a queue that my requests are sitting in before the counter for $request_time begins. – Will Sep 26 '13 at 19:00
  • AFAIK the only thing the can happen before $request_time is OS tcp stack limits, you can try tuning nginx backlog parameter (for listen directive) if this is really the cause. – Andrei Mikhaltsov Sep 26 '13 at 19:15
  • I figured that might be the case. How would I check to see if the kernel if backlogged? – Will Sep 26 '13 at 19:35
0

if you take the highest conncurrent connections during peak-times and multiply this value by by 1.5, can you asure that the connection-pool of your loadbalancer & app-servers are not exhausted? do you monitor app-server-/ha-proxy-response-time? can you asure that your app-servers are not the issue?

that guy from over there
  • 1,355
  • 10
  • 11