HAProxy SSL Connection

Question

We are using HA-Proxy version 1.5.11 2015/01/31, and from yesterday we noticed that all the request to our service over https is very slow. From the Chrome Developer Console, we can see the following timings:

Initial Connection 7.59s
SSL 6.42s

We tested the socket statistics

# ss -s
Total: 2080 (kernel 2088)
TCP: 2674 (estab 2141, closed 433, orphaned 84, synrecv 0, timewait 433/0), ports 0

So it is well below 65k ports that are possible.

Following is our haproxy.cfg

# Global configuration
global
  log 127.0.0.1 local0 notice
  maxconn 50000
  stats socket /tmp/proxystats level admin
  tune.ssl.default-dh-param 2048
  #user deploy
  #group deploy
  #daemon
  tune.bufsize 32768

# Default configuration
defaults
  log global
  mode http
  option httplog
  option dontlognull
  stats enable
  stats uri /proxystats
  stats auth username:pass
  stats realm Haproxy\ Statistics
  stats refresh 5s
  timeout connect 120000
  timeout client 120000
  timeout server 120000
  option redispatch
  option forwardfor
  option http-server-close
  errorfile 500 /etc/haproxy/errors/503.http
  errorfile 502 /etc/haproxy/errors/503.http
  errorfile 503 /etc/haproxy/errors/503.http

# HTTP frontend configuration
frontend http
  mode http
  bind *:80
  #redirect scheme https if !{ ssl_fc }
  redirect prefix https://myservice.com code 301 if { hdr(host) -i myservice.com }
  acl www       hdr(host) -i www.myservice.com
  acl api       hdr(host) -i api.myservice.com
  acl browser   hdr(host) -i br-rx.myservice.com
  use_backend api_server if www
  use_backend api_server if api
  use_backend browser_receiver if browser


# HTTPs frontend configuration
frontend https
  mode http
  bind *:443 ssl crt <our .com pem> crt <second domain pem> crt <third domain pem>
  redirect prefix https://www.myservice.com code 301 if { hdr(host) -i myservice.com }
  use_backend api_server if { ssl_fc_sni www.myservice.com }
  use_backend api_server if { ssl_fc_sni api.myservice.com }
  use_backend browser_receiver if { ssl_fc_sni br-rx.myservice.com }

The CPU and Memory within the system are normal. CPU 9.3%, MEM: 335MB

Where else can we start looking?

What if you lower the `timeout client` from 120 seconds, to something more reasonable for a web request, like 3-5 seconds? — GregL, Oct 21 '15 at 13:54

Willy Tarreau · Accepted Answer · 2015-10-22T11:56:14.450

Given that Chrome reports extremely high connection setup times, it means that SYN packets are being dropped or at least not being answered. This can happen in three situations :

packet losses : you may want to ensure that your internet link is still OK and that this specific server has correct connectivity (its network card could be dead)
backlog full, this happens if haproxy is taking time to accept connections, and results in many SYN_RECV connections, but since you don't have any of them it's not the case ;
improperly tuned conntrack causing incoming connections to be dropped. I would tend to vote for this one given that people deploy load balancers on systems without tuning them and this issue is quite frequent. Please at least check the system's logs using "dmesg" and look for various errors, any net_ratelimit or any "conntrack table full" message.

Edit: I'm just realizing that you only changed the global maxconn setting but not the default one, so your frontends are still limited to 2000 concurrent connections (check with haproxy -vv). And your netstat seems to indicate you're not too far from this limit, so it might be one reason. Please add a maxconn directive in your defaults section.

Thanks Willy for your timely help. The frontend limitation increase solved the issue. — Sundar, Oct 23 '15 at 09:52

HAProxy SSL Connection

1 Answers1