Improving NGINX Throughput on a Single SSL Thread

Question

We are configuring a file delivery server using nginx. The server will be serving large files over HTTPS.

We have run into an issue where we can only achieve around 25MB/s on a single HTTPS thread.

We have tested using a non-HTTPS single download thread (http://) and can achieve full line speed (1Gb/s) at around 120MB/s.

CPU is not anywhere near max encrypting the transfers. We have PLENTY of processing power spare.

We are using aio threads and directio for the file delivery system with large output buffers.

Here is an example of our config:

server {

sendfile off;
directio 512;
aio threads;
output_buffers 1 2m;

            server_name  downloads.oursite.com;
            listen       1.1.1.1:443 ssl;
            ssl_certificate /volume1/Backups/nginxserver/ourdownloads.cer;
            ssl_certificate_key /volume1/Backups/nginxserver/ourdownloads.key;
            ssl on;
            ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
            ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
            ssl_prefer_server_ciphers on;
            ssl_stapling on;
            ssl_stapling_verify on;
            resolver 8.8.4.4 8.8.8.8 valid=300s;
            resolver_timeout 10s;

location = / {
        rewrite ^ https://oursite.com/downloads.html permanent;
}


error_page 404 /404.html;
        location = /404.html {
                root /volume1/Backups/nginxserver/pages/;
                internal;
        }


location / {
                root   /volume1/downloads.oursite.com;
    limit_conn_status 429;
 limit_conn alpha 50;
}
}

Does anybody know how we can achieve faster transfer speeds for a single thread over an SSL connection? What is causing this? Thank you for your tips, suggestions, advice and help in advance.

Are you certain that you are CPU+SSL bound? What throughput do you get without https? Are you also seeing tcp retransmits in `netstat -s` or `nstat` — Aaron, Dec 14 '18 at 15:59
Please see our comment above. We get full line speed on a single thread without HTTPS. Also, look at our answer below - the openssl speed tests show the exact throughput we get on a single HTTPS thread. If we double the threads we get double the throughput. Any ideas? Thanks for your help. — Ted Wilmont, Dec 15 '18 at 16:29

score 0 · Accepted Answer · answered Dec 13 '18 at 14:45

It seems our CPU is to blame. No built-in AES encryption support.

admin@RackStation:/$ openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 5462473 aes-128-cbc's in 2.97s
Doing aes-128-cbc for 3s on 64 size blocks: 1516211 aes-128-cbc's in 2.97s
Doing aes-128-cbc for 3s on 256 size blocks: 392944 aes-128-cbc's in 2.97s
Doing aes-128-cbc for 3s on 1024 size blocks: 98875 aes-128-cbc's in 2.98s
Doing aes-128-cbc for 3s on 8192 size blocks: 12479 aes-128-cbc's in 2.97s
OpenSSL 1.0.2o-fips  27 Mar 2018
built on: reproducible build, date unspecified
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) blowfish(idx) 
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      29427.46k    32672.56k    33869.92k    33975.84k    34420.19k

Improving NGINX Throughput on a Single SSL Thread

1 Answers1