23

I already have HTTPS in place to terminate external HTTPS connections at my AWS ELB. I'm now attempting to secure my connections between my ELB and my backend NGINX servers on EC2 using HTTPS with a self-signed certificate. I've followed the documentation, but accessing the server over HTTPS results in a 408 HTTP timeout. I can't seem to get any debugging information to determine where things are failing.

  • I've confirmed that the security groups permit connections between the ELB and NGINX on EC2.
  • I've convirmed that the VPC allows traffic to be routed between the ELB and EC2 nodes (also HTTP works fine).
  • I've confirmed that the HTTPS listener on the EC2 node is operating (I can hit it directly w/o going to the ELB.
  • I have created an ELB policy of type PublicKeyPolicyType, and associated my public key.
  • I have created an ELB policy of tyep BackendServerAuthenticationPolicyType, and associated it with the PublicKeyPolicyType.
  • I have associated the BackendServerAuthenticationPolicyType with with the ELB.
  • I have ensured that the SSLNegotiationPolicyType supports the algorithms and ciphers I have specified in my NGINX config.
  • I see HTTP requests in my NGINX access logs, but not HTTPS requests.

Is there any way I can get any additional diagnostic information to test this?

Here is my ELB configuration:

$ aws elb describe-load-balancers --load-balancer-name <MY-ELB-NAME>

{
    "LoadBalancerDescriptions": [
        {
            "Subnets": [
                "<REDACTED>",
                "<REDACTED>",
                "<REDACTED>"
            ],
            "CanonicalHostedZoneNameID": "<REDACTED>",
            "VPCId": "<REDACTED>",
            "ListenerDescriptions": [
                {
                    "Listener": {
                        "InstancePort": 80,
                        "LoadBalancerPort": 80,
                        "Protocol": "HTTP",
                        "InstanceProtocol": "HTTP"
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "InstancePort": 443,
                        "SSLCertificateId": "<REDACTED>",
                        "LoadBalancerPort": 443,
                        "Protocol": "HTTPS",
                        "InstanceProtocol": "HTTPS"
                    },
                    "PolicyNames": [
                        "ELBSecurityPolicy-2015-05"
                    ]
                }
            ],
            "HealthCheck": {
                "HealthyThreshold": 2,
                "Interval": 30,
                "Target": "HTTP:80/health",
                "Timeout": 10,
                "UnhealthyThreshold": 2
            },
            "BackendServerDescriptions": [
                {
                    "InstancePort": 443,
                    "PolicyNames": [
                        "MyBackendServerAuthenticationPolicy"
                    ]
                }
            ],
            "Instances": [
                {
                    "InstanceId": "<REDACTED>"
                }
            ],
            "DNSName": "<REDACTED>.us-west-2.elb.amazonaws.com",
            "SecurityGroups": [
                "<GROUP_ID>"
            ],
            "Policies": {
                "LBCookieStickinessPolicies": [],
                "AppCookieStickinessPolicies": [],
                "OtherPolicies": [
                    "ELBSecurityPolicy-2015-05",
                    "MyBackendServerAuthenticationPolicy",
                    "MyPublicKeyPolicy"
                ]
            },
            "LoadBalancerName": "<MY-ELB-NAME>",
            "CreatedTime": "2016-03-23T20:58:49.490Z",
            "AvailabilityZones": [
                "us-west-2a",
                "us-west-2b",
                "us-west-2c"
            ],
            "Scheme": "internal",
            "SourceSecurityGroup": {
                "OwnerAlias": "<REDACTED>",
                "GroupName": "<GROUP_NAME>"
            }
        }
    ]
}

Here are my ELB policies:

$ aws elb describe-load-balancer-policies --load-balancer-name <MY-ELB-NAME>
{
    "PolicyDescriptions": [
        {
            "PolicyAttributeDescriptions": [
                {
                    "AttributeName": "Reference-Security-Policy",
                    "AttributeValue": "ELBSecurityPolicy-2015-05"
                },
                ...
                {
                    "AttributeName": "Protocol-TLSv1.2",
                    "AttributeValue": "true"
                },
                ...
                {
                    "AttributeName": "ECDHE-RSA-AES128-GCM-SHA256",
                    "AttributeValue": "true"
                },
                ...
            ],
            "PolicyName": "ELBSecurityPolicy-2015-05",
            "PolicyTypeName": "SSLNegotiationPolicyType"
        },
        {
            "PolicyAttributeDescriptions": [
                {
                    "AttributeName": "PublicKeyPolicyName",
                    "AttributeValue": "MyPublicKeyPolicy"
                }
            ],
            "PolicyName": "MyBackendServerAuthenticationPolicy",
            "PolicyTypeName": "BackendServerAuthenticationPolicyType"
        },
        {
            "PolicyAttributeDescriptions": [
                {
                    "AttributeName": "PublicKey",
                    "AttributeValue": "<REDACTED>"
                }
            ],
            "PolicyName": "MyPublicKeyPolicy",
            "PolicyTypeName": "PublicKeyPolicyType"
        }
    ]
}

Here is my NGINX config:

worker_processes 10;
worker_rlimit_nofile 8192;
events {
  worker_connections  4096;
}

error_log syslog:server=unix:/dev/log error;
pid       logs/nginx.pid;

http {
  default_type  application/octet-stream;

  log_subrequest on;
  access_log syslog:server=unix:/dev/log,severity=debug extended;

  tcp_nodelay    on;
  tcp_nopush     on;

  server_tokens off;

  upstream api {
    server localhost:8080;
  }

  server {
    listen 80 default_server;
    listen [::]:80 default_server;

    location / {
      # Redirect all other HTTP requests to HTTPS with a 301 Moved Permanently response.
      return 301 https://$host$request_uri;
    }
  }

  server {
    listen 443 ssl;
    listen [::]:443 ssl;

    ssl_certificate /path/to/ssl.crt;
    ssl_certificate_key /path/to/ssl.key;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;ECDHE

    # Diffie-Hellman parameter for DHE ciphersuites, recommended 2048 bits
    ssl_dhparam /path/to/dhparam.pem;

    # modern configuration. tweak to your needs.
    # See: https://mozilla.github.io/server-side-tls/ssl-config-generator/
    ssl_protocols TLSv1.2;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
    ssl_prefer_server_ciphers on;

    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains;";

    # Our main location to proxy everything else to the upstream
    # server, but with the added logic for enforcing HTTPS.
    location / {
      proxy_http_version 1.1;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect off;
      proxy_next_upstream error;

      proxy_pass http://api;
    }
  }
}

I'm generating keys/certificates using the following commands:

$ openssl genrsa \
  -out /path/to/ssl.key 2048
$ openssl req \
  -sha256 \
  -new \
  -key /path/to/ssl.key \
  -out /path/to/ssl.csr
$ openssl x509 \
  -req \
  -days 365 \
  -in /path/to/ssl.csr \
  -signkey /path/to/ssl.key \
  -out /path/to/ssl.crt
$ openssl dhparam -out /path/to/dhparam.pem 2048
jsears
  • 4,511
  • 2
  • 31
  • 36
  • 2
    Are you doing this for compliance reasons (PCI/HIPAA/etc), or just because you think you need it? The network traffic between the ELB and the web servers will be contained within your VPC, so it would only be compromised if an attacker gained access to a server within your VPC. SSL to the ELB is generally considered good enough unless you have legal requirements for "encryption in motion". – Mark B Mar 23 '16 at 15:42
  • 2
    Yes, we have legal requirements to encrypt traffic that goes over the wire, that and it's a good practice. – jsears Mar 23 '16 at 16:26
  • Is the `Instance Protocol` setting for the listener set to `HTTPS` (in the console)? HTTPS and 443 need to be set on the left side and right side on that screen, and the right side default is HTTP ... I'm at a loss as to what would cause a 408 if not this. – Michael - sqlbot Mar 24 '16 at 00:18
  • @Michael-sqlbot Sorry, I should have posted my ELB Listener settings. I'll update my question with that info. To answer your question, yes, the Instance Protocol is set to HTTPS and it is forwarding to 443 on the backend instance. – jsears Mar 24 '16 at 13:22
  • @racl101 This should probably be a different SO question. The comments describe what the "location /" section does. The first one redirects HTTP requests to the HTTPS endpoint. The second one proxies requests to the app-server. – jsears Nov 14 '17 at 14:41
  • To do this with an ALB, see https://stackoverflow.com/questions/54997493/elb-to-backend-server-using-https-with-self-signed-certificate/55274301#55274301 – Charles L. Mar 21 '19 at 05:33

1 Answers1

9

Adding some non-EC DHE ciphers to the NGINX config solved this for me. I've switched to the following config in the HTTPS listener in nginx.conf:

  # intermediate configuration. tweak to your needs.
  ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';

I'd like to drop all non-EC DHE ciphers and only support ECDHE. I suspect this fixes the problem because I'm generating an RSA key/cert instead of an EC key/cert. If anyone knows how I can properly generate an EC key/cert, and then correctly extract the EC public key for upload to AWS, please improve upon my answer. I've attempted to generate an EC key/cert, but when I try to create the ELB public key policy, AWS reports it as an invalid public key.

jsears
  • 4,511
  • 2
  • 31
  • 36
  • 1
    Could you update your post to include the instructions/commands you used for creating and uploading your EC key/cert, and the exact error/response you get back from AWS when attempting to upload that EC key/cert? – Castaglia Mar 24 '16 at 19:15
  • 1
    This worked for me. I was previously using the "Mozilla Modern" SSL configuration for nginx and upon switching to "Intermediate" everything just worked. All the modern ciphers start with "ECDHE-" while intermediate includes some "DHE-", "AES128-", "AES256-" and "EDH-". I haven't dug deeper to see which AWS ELBs are actually using. – notpeter Jul 06 '17 at 19:14