0

I have been facing issues on my app under load and after running some load tests, I realised that when I directly load test the app server over http on port 8080, the app server responds fairly quickly.

When I load test by hitting the domain name and using https, i.e. the endoint that I hit is https://load.api.example.in, the response time increases about 5x.

This leads me to the conclusion that the haproxy in between is a bottleneck.

I increased the server size for haproxy, and under heavy load, CPU utilization spikes, but because of the larger server is only about 50%. (as per AWS monitoring)

What do I need to change in the haproxy config (or anywhere else) to fix this issue?

My present haproxy config (anonymized)

global
        ulimit-n 99999
        maxconn 99999
        maxpipes 99999
        log 127.0.0.1 local0
        log 127.0.0.1 local1 notice
        maxconn 4096
        tune.ssl.default-dh-param 2048
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        daemon

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option forwardfor
        option http-server-close
        retries 3
        option redispatch
        maxconn 2000
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend mqtt
        bind *:1883
        bind *:1884
        mode tcp
        option clitcpka # TCP Keep-alive
        timeout client 3h
        timeout server 3h
        option tcplog

        acl host_mqtt_staging hdr(host) -i staging.mqtt.example.in
        acl host_mqtt_staging hdr(host) -i staging.mqtt.example.com
        acl host_mqtt hdr(host) -i mqtt.example.in
        acl host_mqtt hdr(host) -i mqtt.example.com
        use_backend mqtt_staging if host_mqtt_staging
        use_backend mqtt if host_mqtt

frontend http
        bind *:80
        mode http
        reqadd X-Forwarded-Proto:\ http
        redirect scheme https code 301 if !{ ssl_fc }

frontend https
        bind *:443 ssl crt /etc/haproxy/certs/staging.myservice.example-2.com.pem
        mode http
        reqadd X-Forwarded-Proto:\ https

        acl letsencrypt-acl path_beg /.well-known/acme-challenge/
        acl host_myservice_staging hdr(host) -i staging.api.example.in
        acl host_myservice_staging hdr(host) -i staging.api.example.com
        acl host_myservice_load hdr(host) -i load.api.example.in

        use_backend letsencrypt-backend if letsencrypt-acl
        use_backend myservice_staging if host_myservice_staging
        use_backend myservice_load if host_myservice_load
        default_backend api

backend letsencrypt-backend
        server letsencrypt 127.0.0.1:54321

backend api
        balance roundrobin
        option httpclose
        option forwardfor
        server web00 app00.staging.internal.example-2.com:8080 check

backend myservice_staging
        balance roundrobin
        option httpclose
        option forwardfor
        server myservice00 myservice00.staging.internal.example-2.com:8080 check weight 1

backend myservice_load
        balance roundrobin
        option httpclose
        option forwardfor
        server myserviceload00 load00.staging.internal.example-2.com:8080 check weight 1

backend mqtt_staging
        balance leastconn
        server mqttdev00 mqtt00.internal.example-2.com:1883 check

backend mqtt
        balance leastconn
        server prodmqttdev00 prodmqtt00.internal.example-2.com:1883 check

Response times

  • With haproxy: Avg of over 3 seconds
  • without haproxy: Avg of between 500-600 ms

Update: While testing, I updated my haproxy config, to forward http to the same backends as is already configured in https. With this change, my load test (via haproxy on http) performed as well as the load test where I sent all requests directly to the server.
Therefore, I'm pretty certain that the issue (or the biggest issue of several issues) is the ssl configuration in haproxy. Any suggestions on what I should try changing to improve performance?

kapad
  • 113
  • 6
  • response times: with haproxy: over 3 seconds average without haproxy: between 500-600 ms average. – kapad Feb 21 '18 at 07:36
  • try to test haproxy with only http, if everything works like that, maybe you have problem with letsencrypt – c4f4t0r Feb 21 '18 at 13:02
  • I did test haproxy with only http.. that worked as well as going to the server directly. have added an update with this info. – kapad Feb 21 '18 at 13:12
  • What is the latency from the client to HaProxy? is it the 500ms or is the backing application slow/adding time? – Brennen Smith Mar 02 '18 at 21:59
  • No.. the backend application is quite fast. I see it has an average reponse time of less than 700ms. So it's definitely haproxy where the latency is being added. – kapad Mar 04 '18 at 07:44

1 Answers1

1

As you are already using AWS, use ELB+AWS Certificate Manager to do SSL for you. Run Haproxy in http mode and ignore the issue.

Or start playing with the following Haproxy options:

tune.ssl.cachesize
tune.ssl.lifetime
ciphers
defer-accept
force-tlsv12

Also try to use a static page on the backend, preferably on the same host. To limit the applications impact during testing.

Gothrek
  • 531
  • 2
  • 8
  • thanks for this. through my searching online I'd already found and tuned the cachesize and the lifetime. what do the other options affect? and more importantly, are there any concerns with changing them (security wise concerns)? especially with the ciphers and force-tlsv12 options. – kapad Mar 04 '18 at 21:03
  • 1
    Ciphers parameter defines supported ciphers for ssl negotiation and their order. Defer-accept only accepts connections after some data arrives on them. This partially mitigates slowloris attacks. force-tlsv12 enforces the usage of tls v1.2 in ssl. – Gothrek Mar 05 '18 at 09:04