NGINX SSL Reverse Proxy Verify Upstream SSL

Question

I have NGINX setup as a reverse proxy to host multiple websites using only one IP address. I have a Lets Encrypt certificate on the proxy and a different Lets Encrypt certificate on the upstream server. Basically, I need NGINX to forward traffic to the upstream server and verify that the upstream server has a valid TLS certificate. If I disable proxy_ssl_verify, it will work. If I go to https://app.local.example.com on my internal network, the app works fine.

Network Diagram:

https://app.example.com --> NGINX Reverse Proxy --> https://app.local.example.com (Local IP Address)

NGINX Reverse Proxy Configuration File:

server {
    listen 80;
    rewrite ^ https://$host$request_uri? permanent;
}

server {
        server_name app.example.com;

        listen                                  443 ssl;

        ssl_certificate                         /etc/letsencrypt/live/app.example.com/cert.pem;
        ssl_certificate_key                     /etc/letsencrypt/live/app.example.com/privkey.pem;
        ssl_trusted_certificate                 /etc/letsencrypt/live/app.example.com/chain.pem;

        location / {

                proxy_redirect off;
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Ssl on;
                proxy_set_header X-Forwarded-Protocol $scheme;
                proxy_set_header X-Forwarded-HTTPS on;

                proxy_ssl_session_reuse        off;
                proxy_ssl_name                 app.local.example.com

                proxy_ssl_verify               on;
                proxy_ssl_verify_depth         2; # I've tried 1,2,3,4
                proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt; 
                            
                proxy_pass                     https://app.local.example.com
        }
}

Here's the error message I'm getting.

[error] 1087#1087: *2 upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream, client: [Client IP], server: app.example.com, request: "GET / HTTP/1.1", upstream: "https://192.168.1.5:443/", host: "app.local.example.com", referrer: "https://app.example.com/">

OpenSSL Version: OpenSSL 1.1.1f 31 Mar 2020

nginx -v: nginx version: nginx/1.18.0 (Ubuntu)

cat /etc/issue: Ubuntu 20.04.1 LTS \n \l

Output of openssl s_client -connect app.local.example.com:443

CONNECTED(00000003)
depth=0 CN = app.local.example.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = app.local.example.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:CN = app.local.example.com
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFeDCCBGCgAwIBAgISA8NkbZ6wz2EnKcedXujKT9AmMA0GCSqGSIb3DQEBCwUA
...
-----END CERTIFICATE-----
...
SSL-Session:
    Protocol  : TLSv1.3
    ...
    Verify return code: 21 (unable to verify the first certificate)

*"If I go to `https://app.local.example.com` on my internal network, the app works fine."* - browsers tend to work around badly configured servers with missing intermediate certificates. Could you do a `openssl s_client -connect app.local.example.com:443` to access the internal site instead of a browser and check the verification status. Maybe just add the relevant parts of the output into your question, especially the part where it shows which certificate chain is provided by the server. — Steffen Ullrich, Aug 17 '20 at 19:07

score 2 · Answer 1 · answered Aug 17 '20 at 19:42

Certificate chain
 0 s:CN = app.local.example.com
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

Your server is not properly configured. Instead of sending leaf certificate (app.local.example.com) and intermediate certificate it only sends the leaf certificate. Because of the missing intermediate certificate no trust path can be created to the local trust anchor, which means the certificate validation fails with "unable to get local issuer certificate". Browsers often work around such problems by getting the missing intermediate certificate from somewhere else, but other TLS stacks usually don't do such workarounds.

Once you've properly configured the server you should see the following in openssl s_client. Once this is done nginx should work too.

Certificate chain
 0 s:CN = app.local.example.com
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

score 2 · Accepted Answer · answered Aug 18 '20 at 06:17

I changed the ssl_certificate value from cert.pem to fullchain.pem on the upstream server.

ssl_certificate         /etc/letsencrypt/live/app.example.com/fullchain.pem;
ssl_certificate_key     /etc/letsencrypt/live/app.example.com/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/app.example.com/chain.pem;

Here's a link to NGINX config files I used

NGINX SSL Reverse Proxy Verify Upstream SSL

2 Answers2