Unable to use LetsEncrypt - CertBot - When HTTP to HTTPS redirect is setup

Question

I am trying to configure CertBot and it only works when I serve my site over http. Usually I have an https redirect and I don't want to have to change the site config each time I need to use certbot. I tried to serve only /.well-known/ over http but it is still failing any ideas how to resolve this?

I am trying to copy this idea but not working --> NGINX redirect everything except letsencrypt to https

Eg: This Works:

server {
    listen 80;
    listen [::]:80;
    server_name example.com www.example.com;


location / {

        proxy_pass              http://localhost:8575/;
        include                 /etc/nginx/conf.d/proxy.conf;
    }
}

This does not: (Note that the current configured SSL Certs are not correct, but needed for NGinX to start)

server {

   listen 80;
   listen [::]:80;
   server_name www.example.com example.com;

    location /.well-known/acme-challenge/ {
        proxy_pass              http://localhost:8575/;
        include                 /etc/nginx/conf.d/proxy.conf;
    }

location / {
       return 301 https://$server_name$request_uri;
    }

}

server {
        listen 443 ssl;
        listen        [::]:443;
        server_name www.example.com example.com;

#        ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
#        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        ssl_certificate /etc/ssl/crt/crt.crt;
        ssl_certificate_key /etc/ssl/crt/key.key;

location / {

        proxy_pass              http://localhost:8575/;
        include                 /etc/nginx/conf.d/proxy.conf;
    }
}

Error Log:

certbot    | Saving debug log to /var/log/letsencrypt/letsencrypt.log
certbot    | Plugins selected: Authenticator webroot, Installer None
certbot    | Registering without email!
certbot    | Obtaining a new certificate
certbot    | Performing the following challenges:
certbot    | http-01 challenge for www.example.com
certbot    | http-01 challenge for example.com
certbot    | Using the webroot path /var/www/html for all unmatched domains.
certbot    | Waiting for verification...
certbot    | Challenge failed for domain www.example.com
certbot    | Challenge failed for domain example.com
certbot    | http-01 challenge for www.example.com
certbot    | http-01 challenge for example.com
certbot    | Cleaning up challenges
certbot    | IMPORTANT NOTES:
certbot    |  - The following errors were reported by the server:
certbot    |
certbot    |    Domain: www.example.com
certbot    |    Type:   unauthorized
certbot    |    Detail: Invalid response from
certbot    |    http://www.example.com/.well-known/acme-challenge/WyVEA5g6BWVDPpYUhEJ0bG5iH6daF1rZpFd0vuTXOa0
certbot    |    [50.117.156.123]: "        <!DOCTYPE html><html lang=\"en-US\">\r\n
certbot    |    \t<head>\n\n\t\t        <meta charset=\"UTF-8\">\r\n        <meta
certbot    |    name=\"viewport\" con"
certbot    |
certbot    |    Domain: example.com
certbot    |    Type:   unauthorized
certbot    |    Detail: Invalid response from
certbot    |    https://www.example.com/x61_h9wxFY2Ye8-16GllyMq_dfsXbsEB1lYOjeq4LjU
certbot    |    [50.117.156.123]: "        <!DOCTYPE html><html lang=\"en-US\">\r\n
certbot    |    \t<head>\n\n\t\t        <meta charset=\"UTF-8\">\r\n        <meta
certbot    |    name=\"viewport\" con"
certbot    |
certbot    |    To fix these errors, please make sure that your domain name was
certbot    |    entered correctly and the DNS A/AAAA record(s) for that domain
certbot    |    contain(s) the right IP address.
certbot    |  - Your account credentials have been saved in your Certbot
certbot    |    configuration directory at /etc/letsencrypt. You should make a
certbot    |    secure backup of this folder now. This configuration directory will
certbot    |    also contain certificates and private keys obtained by Certbot so
certbot    |    making regular backups of this folder is ideal.
certbot    | Some challenges have failed.
certbot exited with code 1

Related: https://serverfault.com/questions/1015642/let-s-encrypt-certbot-how-to-use-https-for-the-server-validation — Jesse Nickles, Dec 11 '22 at 18:19

Esa Jokinen · Accepted Answer · 2020-05-22T09:31:49.280

4

With HTTP-01 challenge it's OK to redirect to HTTPS first and serve the challenge over TLS. However, the challenge always starts with a plain HTTP connection using port 80, and you can only redirect to HTTPS on port 443.

Our implementation of the HTTP-01 challenge follows redirects, up to 10 redirects deep. It only accepts redirects to “http:” or “https:”, and only to ports 80 or 443. It does not accept redirects to IP addresses. When redirected to an HTTPS URL, it does not validate certificates (since this challenge is intended to bootstrap valid certificates, it may encounter self-signed or expired certificates along the way).

The HTTP-01 challenge can only be done on port 80. Allowing clients to specify arbitrary ports would make the challenge less secure, and so it is not allowed by the ACME standard.

Therefore, this kind of Nginx configuration should work, as well:

server {
    listen 80;
    server_name www.example.com example.com;

    location / {
        return 301 https://$server_name$request_uri;
    }
}

server {
    listen 443 ssl;
    server_name www.example.com example.com;

    location /.well-known/acme-challenge/ {
        # HTTP-01 challenge
    }

    location / {
        # Your web application
    }
}

In your case this means the following could be either in the HTTP or the HTTPS server block.

location /.well-known/acme-challenge/ {
    proxy_pass   http://localhost:8575/.well-known/acme-challenge/;
    include      /etc/nginx/conf.d/proxy.conf;
}

You were able to replace the /.well-known/acme-challenge/ with $request_uri because:

When variables are used in proxy_pass:
location /name/ {
    proxy_pass http://127.0.0.1$request_uri;
}
In this case, if URI is specified in the directive, it is passed to the server as is, replacing the original request URI.

Also, if the / has the same root, you don't need a separate location at all.

edited May 22 '20 at 09:31

answered May 22 '20 at 08:10

Esa Jokinen

46,944
3
83
129

Just don't understand why it needs its own server location when root is forwarded normally perhaps if I tried adding requesturi with just my normal https root redirect it wouldn't affect anything and allow this to work with my configurations as they are. Perhaps it should be added to all my configurations – FreeSoftwareServers May 22 '20 at 08:14
If it's served from the same root or using the same proxy, it should work without the separate `location` block. This is a more generalized answer for serving it differently, if e.g. the web application was reverse proxied and `/acme-challenge/` served as static files. – Esa Jokinen May 22 '20 at 08:17
OK, but this should simplify my setup too the real trick for me was requestURI inside the proxy_pass directive, I'll go remove the location block tomorrow and retest. – FreeSoftwareServers May 22 '20 at 08:19
That was because you were proxying `location /.well-known/acme-challenge/` to `http://localhost:8575/` instead of `http://localhost:8575/.well-known/acme-challenge/`. – Esa Jokinen May 22 '20 at 08:22
Kinda figured it would just proxy all that be default. I've never had to add anything to the proxy directive beside IP:PORT for anything in the past. – FreeSoftwareServers May 22 '20 at 08:24
1

Think that you had a web application on `http://localhost:8080/` (not in `http://localhost:8080/app/`) and you wanted to serve it on `https://example.com/app/`, because the root was used for another website. That wouldn't be possible, if the proxy automatically added the `location` path to the request. – Esa Jokinen May 22 '20 at 08:35
I'm thinking about incorporating yours and Ajays answer as well. See nginx doesn't have access to the websites webroot, but really, it doesn't need it. It just needs access to "any directory" and then serve that directory in a separate location block without `proxy_pass` and without `request_uri`. Only thing I'm not sure of, if I had multiple websites, would they each need a separate folder for the acme challenge or can I just have nginx have one folder it uses for certbot that is local. The IP already resolves to that nginx server setup on 80 & 443. – FreeSoftwareServers May 22 '20 at 08:41
1

The webroot for ACME challenge can be same for every domain. With Apache it's even possible to add a global alias, which makes it a lot easier to add new domains even before they are configured. In Nginx you could have this served from the default server. – Esa Jokinen May 22 '20 at 09:13
I love the "Default Server!~" idea. I do have one setup to make sure incorrect domains aren't served. Thanks for the tip! – FreeSoftwareServers May 22 '20 at 09:21
Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/108328/discussion-between-esa-jokinen-and-freesoftwareservers). – Esa Jokinen May 22 '20 at 09:26
I'm unable to test at the moment and happy with the information I have. I beleive this is all relative to the post, if you wanted to incorporate into post I can clean up my comments. I may even switch answers after I get it all setup as "default_server" sounds ideal. – FreeSoftwareServers May 22 '20 at 09:39

FreeSoftwareServers · Answer 2 · 2020-05-22T10:42:31.027

Update: My main issue was not using $request_uri w/ proxy_pass directive (which also doesn't allow ~ BTW). But, there was nothing wrong with using this in the HTTPS block. Further more after looking at both https://serverfault.com/a/1018199/312793 and

https://serverfault.com/a/1017720/312793 I realized that I don't need to actually pass my "real" root directory of my webapp, just somewhere that nginx can serve to certbot to read/write files. As well, you can have one directory serve multiple sites so I decided it would be most proficient to add the location inside the default nginx server block I have setup to re-route incorrectly formatted requests to include certbot so I can now add domains without adjust any configs 100%. In fact, the web app doesn't even need to be running, just nginx.

Here is my new default server block. Note: I created a folder acme inside my nginx "real" webroot and serve that directory for the location /.well-known/acme-challenge/

server {
        listen 80 default_server;
        listen [::]:80 default_server;
        root /var/www/html/;
        index index.html;
}
server {
        listen 443 default_server;
        listen [::]:443 default_server;
        ssl_certificate /etc/ssl/fake/fake.crt;
        ssl_certificate_key /etc/ssl/fake/fake.key;

        location /.well-known/acme-challenge/ {
        root /var/www/html/acme;
         allow all;
        }

        root /var/www/html/;
        index index.html;
}

Just like when doing setup, you need to have something for SSL certs or nginx won't start correctly. Very happy with this setup/resolution!

Needed to add the following: $request_uri

location /.well-known/acme-challenge/ {
    proxy_pass              http://localhost:8575/$request_uri;
    include                 /etc/nginx/conf.d/proxy.conf;
}

Great that you were able to solve it. Please accept your answer, otherwise the question will not be marked as "solved" in the system and will pop up again and again in the future. It is perfectly fine to accept your own answers. — Gerald Schneider, May 19 '20 at 06:52

Unable to use LetsEncrypt - CertBot - When HTTP to HTTPS redirect is setup

2 Answers2

Linked