nginx config using variable in ssl_certificate path throws permissions error

Question

The nginx configuration server block:

localhost:/etc/nginx$ cat nginx.conf | grep -B 3 -A 6 '$ssl_server_name'
  server {
    listen 443 ssl http2 default_server;

    ssl_certificate         /etc/letsencrypt/live/$ssl_server_name/fullchain.pem;
    ssl_certificate_key     /etc/letsencrypt/live/$ssl_server_name/privkey.pem;

    location / {
      include /etc/nginx/snippets/set-headers.conf;
      proxy_pass http://localhost:8080;
    }
  }

This is using the variable $ssl_server_name in the certificate directives which is supported since nginx 1.15.9. Relevant part of the nginx docs.

The configuration passes nginx -t and loads without issues, but page does not load in browser, and there is a permissions denied error opening the cert in error.log even though nginx is running as root:

localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/19 18:51:47 [error] 5676#5676: *251 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443
localhost:/etc/nginx$ ps -ef | grep nginx | grep -v grep
www-data  5676 24653  0 18:49 ?        00:00:00 nginx: worker process
root     24653     1  0 15:08 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem
lrwxrwxrwx 1 root root 56 Apr 17 18:53 /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem -> ../../archive/[DOMAIN NAME REDACTED]/fullchain1.pem
localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
-rw-r--r-- 1 root root 3591 Apr 17 18:53 /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/19 18:51:47 [error] 5676#5676: *251 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443
localhost:/etc/nginx$ ps -ef | grep nginx | grep -v grep
www-data  5676 24653  0 18:49 ?        00:00:00 nginx: worker process
root     24653     1  0 15:08 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem
lrwxrwxrwx 1 root root 56 Apr 17 18:53 /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem -> ../../archive/[DOMAIN NAME REDACTED]/fullchain1.pem
localhost:/etc/nginx$ sudo ls -l /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
-rw-r--r-- 1 root root 3591 Apr 17 18:53 /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
localhost:/etc/nginx$ openssl
OpenSSL> version
OpenSSL 1.0.2g  1 Mar 2016
OpenSSL> ^C
localhost:/etc/nginx$ nginx -v
nginx version: nginx/1.17.0

When I replace $ssl_server_name with the domain name in the nginx configuration then there is no permissions error reading the very same cert file, and the page loads in the browser.

Why does using the variable in the cert path not work?

UPDATE:

I updated the archive folder group to www-data, still seing the permissions error:

localhost:/etc/nginx$ sudo chgrp -R www-data /etc/letsencrypt/archive
localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
f: /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
drwxr-xr-x root root     /
drwxr-xr-x root root     etc
drwxr-xr-x root root     letsencrypt
drwx------ root www-data archive
drwxr-xr-x root www-data [DOMAIN NAME REDACTED]
-rw-r--r-- root www-data fullchain1.pem
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 07:18:58 [error] 4897#4897: *6 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

UPDATE 2:

Added group read and execute permissions to archive folder, still seing the permissions error:

localhost:/etc/nginx$ sudo chmod g+r /etc/letsencrypt/archive
localhost:/etc/nginx$ sudo chmod g+x /etc/letsencrypt/archive
localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/archive/ [DOMAIN NAME REDACTED]/fullchain1.pem
f: /etc/letsencrypt/archive/[DOMAIN NAME REDACTED]/fullchain1.pem
drwxr-xr-x root root     /
drwxr-xr-x root root     etc
drwxr-xr-x root root     letsencrypt
drwxr-x--- root www-data archive
drwxr-xr-x root www-data  [DOMAIN NAME REDACTED]
-rw-r--r-- root www-data fullchain1.pem
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 07:39:58 [error] 4897#4897: *22 cannot load certificate "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/ [DOMAIN NAME REDACTED]/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

UPDATE 3:

Tried becoming www-data using sudo but got an error:

localhost:/etc/nginx$ sudo su - www-data
No directory, logging in with HOME=/
This account is currently not available.

Update 4:

I also updated the permissions on the symlinked path live folder, still seing the permissions error:

localhost:/etc/nginx$ ll /etc/letsencrypt | grep live
drwx------   5 root root     4096 Apr 17 18:53 live/
localhost:/etc/nginx$ sudo chgrp www-data /etc/letsencrypt/live
localhost:/etc/nginx$ sudo chmod g+rx /etc/letsencrypt/live
localhost:/etc/nginx$ ll /etc/letsencrypt | grep live
drwxr-x---   5 root www-data 4096 Apr 17 18:53 live/
localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/live
f: /etc/letsencrypt/live
drwxr-xr-x root root     /
drwxr-xr-x root root     etc
drwxr-xr-x root root     letsencrypt
drwxr-x--- root www-data live
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 07:57:48 [error] 5104#5104: *17 cannot load certificate key "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

Update 5:

Listing the permissions of all dirs in path including symlinks:

localhost:/etc/nginx$ sudo namei -l /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem
f: /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem
drwxr-xr-x root root     /
drwxr-xr-x root root     etc
drwxr-xr-x root root     letsencrypt
drwxr-x--- root www-data live
drwxr-xr-x root root     [DOMAIN NAME REDACTED]
lrwxrwxrwx root root     fullchain.pem -> ../../archive/[DOMAIN NAME REDACTED]/fullchain1.pem
drwxr-x--- root www-data   ..
drwxr-xr-x root root       ..
drwxr-x--- root www-data   archive
drwxr-xr-x root www-data   [DOMAIN NAME REDACTED]
-rw-r--r-- root www-data   fullchain1.pem

Update 6:

Tried temporarily changing the shell for www-data user, became www-data using sudo and tested reading the cert was possible, but the permission error is still happening:

localhost:/etc/nginx$ cat /etc/passwd | grep www-data
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
localhost:/$ cat /etc/passwd | grep www-data
www-data:x:33:33:www-data:/var/www:/bin/bash
localhost:/etc/nginx$ sudo vim /etc/passwd
localhost:/etc/nginx$ sudo su - www-data
No directory, logging in with HOME=/
localhost:01:/$ whoami
www-data
localhost:/$ cat /etc/letsencrypt/live/[DOMAIN NAME REDACTED]/fullchain.pem
-----BEGIN CERTIFICATE-----
[REDACTED CERT]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[REDACTED CERT]
-----END CERTIFICATE-----
localhost:/$ exit
logout
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 08:40:23 [error] 5259#5259: *14 cannot load certificate key "/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/[DOMAIN NAME REDACTED]/privkey.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

Update 7:

Tried exporting the certs to another folder:

localhost:/etc/nginx$ mkdir /tmp/exported-certs
localhost:/etc/nginx$ sudo rsync -razL /etc/letsencrypt/live/ /tmp/exported-certs
localhost:/etc/nginx$ sudo ls -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem
-rw-r--r-- 1 root www-data 3591 Apr 17 18:53 /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem
localhost:/etc/nginx$ sudo ls -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
-rw------- 1 root www-data 1704 Apr 17 18:53 /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
localhost:/etc/nginx$ sudo namei -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem
f: /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem
drwxr-xr-x root root     /
drwxrwxrwt root root     tmp
drwxr-x--- root www-data exported-certs
drwxr-xr-x root root     [DOMAIN NAME REDACTED]
-rw-r--r-- root www-data fullchain.pem
localhost:/etc/nginx$ sudo vim nginx.conf
localhost:/etc/nginx$ cat nginx.conf | grep -B 3 -A 6 '$ssl_server_name'
  server {
    listen 443 ssl http2 default_server;

    ssl_certificate /tmp/exported-certs/$ssl_server_name/fullchain.pem;
    ssl_certificate_key /tmp/exported-certs/$ssl_server_name/privkey.pem;

    location / {
      include /etc/nginx/snippets/set-headers.conf;
      proxy_pass http://localhost:8080;
    }
  }
localhost:/etc/nginx$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
localhost:/etc/nginx$ sudo nginx -s reload
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 10:52:48 [notice] 6250#6250: signal process started
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/error.log
2019/06/20 10:53:08 [error] 6251#6251: *67 cannot load certificate key "/tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib) while SSL handshaking, client: [IP ADDRESS REDACTED], server: 0.0.0.0:443

Then decided to check again as the www-data user because last time I checked it was when the certs were in the letsencrypt folder, also this time I remembered to check both cert and key:

localhost:/etc/nginx$ cat /etc/passwd | grep www-data
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
localhost:/etc/nginx$ sudo vim /etc/passwd
localhost:/etc/nginx$ cat /etc/passwd | grep www-data
www-data:x:33:33:www-data:/var/www:/bin/bash
localhost:/etc/nginx$ sudo su - www-data
No directory, logging in with HOME=/
localhost:/$ cat /tmp/exported-certs/[DOMAIN NAME REDACTED]/fullchain.pem
-----BEGIN CERTIFICATE-----
[CERT REDACTED]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[CERT REDACTED]
-----END CERTIFICATE-----
localhost:/$ cat /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
cat: /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem: Permission denied   <---- THERE IT IS!
localhost:/$ ls -l /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
-rw------- 1 root www-data 1704 Apr 17 18:53 /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
localhost:/$ exit
logout
localhost:/etc/nginx$ sudo chmod g+r /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
localhost:/etc/nginx$ sudo su - www-data
No directory, logging in with HOME=/
localhost:/$ cat /tmp/exported-certs/[DOMAIN NAME REDACTED]/privkey.pem
-----BEGIN PRIVATE KEY-----
[CERT REDACTED]
-----END PRIVATE KEY-----
localhost:/$ exit
logout
localhost:/etc/nginx$ sudo tail -n 1 /var/log/nginx/access.log
139.162.202.226 - [DOMAIN NAME REDACTED]:443 - [20/Jun/2019:11:04:08 +0100] "GET / HTTP/2.0" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15"

Once I addd the group read permission for www-data to the privkey.pem, the browser was able to load the page. :)

Thanks to all that commented on this question.

When you provide full absolute name this can be resolved when nginx starts, and it has `root` credentials then, so can load everything. If you provide a path with a variable, the loading can happen only later, when the variable is resolved that is when the connection is made and then nginx has only `www-data` credentials, so the certificate needs to be readable by that user, including all elements of the path leading to it. It is written in the documentation you link to: "Give read-access to the certificates" — Patrick Mevzek, Jun 19 '19 at 19:31
@PatrickMevzek Yes I read that but though that nginx had root permissions or that it read all the certs at startup time, thanks for the details I will try to add read permissions to www-data — vy218, Jun 20 '19 at 04:59
Certificates can appear on disk after the process started... As you can see from your `ps` output, there is a master process running as `root` but another process deals with HTTP exchanges and it runs as `www-data`... — Patrick Mevzek, Jun 20 '19 at 05:23
@PatrickMevzek I updated the group of all files in the archive folder, but I am still seing the same permissions error, see the updated question with the console output. Shouldn't the worker process now be able to read the files? — vy218, Jun 20 '19 at 06:32
You need to check all directories in the path, starting from /etc, www-data must have rx on all of them. Use sudo to become `www-data` momuntarily and try accessing the files — Patrick Mevzek, Jun 20 '19 at 06:37
I think I figured it out, I need to add group read permissions to the archive folder as well... — vy218, Jun 20 '19 at 06:38
@PatrickMevzek I tried becoming www-data user using sudo but that errored, is there another way to verify the permissions setting? — vy218, Jun 20 '19 at 06:49
file system permission is one thing - here you can set even ACL (additional permission next to standard user,group,other). Other thing is additional security layer - SElinux (in "RedHat" line or AppArmor in se of "SuSE" line). Try to check also audit log - ```/var/log/audit/audit.log```. Next what you can check is context ```ps -efZ``` and ```ls -Z```. — Kamil J, Jun 20 '19 at 06:54
@MichaelHampton Thanks I updated as per your suggestion, still seing the permissions error though — vy218, Jun 20 '19 at 07:03
@KamilJ There is no file /var/log/audit/audit.log, it is Ubuntu. ps -efZ was the same as regular ps but said 'unconfined' in column 1. ls -Z on the certs directories just listed the files with ? added before each file. Did you mean some other way to use those commands? — vy218, Jun 20 '19 at 07:07
hmm, you can try also ```sestatus``` and make a focus to line "Current mode:". In case there is not "enforcing" it would not be the case for you. In case of "enforcing" state we can try to continue in this idea thread ;-). — Kamil J, Jun 20 '19 at 07:18
I have updated with listing of permissions of all folders in path including the symlinked folders. All root.root folders have r-x for other, and the rest are root.www-data and have r-x for group, so all permissions are correct in the path as far as I can see, but the error is still happening. — vy218, Jun 20 '19 at 07:23
@KamilJ sestatus is not installed, I don't think ubuntu uses selinux by default, and I haven't set it up — vy218, Jun 20 '19 at 07:25
I was able to test reading the file as www-data user and the user can indeed read the file (see udpate 6). Is the issue then not file permissions? — vy218, Jun 20 '19 at 07:43

score 0 · Answer 1 · answered Jun 20 '19 at 07:40

0

OK, thanks for feedback in comment. Let me share two ideas - not directly solution on what you have mentioned but possible workarounds....

prepare instead of linking

I am using in some specific case certbot (I guess the same as you) and haproxy. I have cron job running certbot with certonly and in case the cert is issued it is concatenating cert and key (one file is preferred) and copy out to location where haproxy expect it in configuration. The haproxy is restarted afterwards. In case you will not solve it and need to have it working may be try to "prepare" the cert for nginx somewhere instead of linking to the "original" location...

run certbot as www-data user

you can run certbot as user who is running nginx (www-data) so the output would fit for the purpose by the permission point of view. In that case you need to change the ownership first for the structure around - e.g. /etc/letsencrypt/account....

answered Jun 20 '19 at 07:40

Kamil J

1,632
1
5
10

Thanks for the suggestions - Not to keen to run certbot as a different use in case it breaks something in how certbot works. The prepare route sounds like it could be a good way to go, I wanted to use the default certbot setup because otherwise I have to figure out how to manage another folder full of certs, but it might not be as bad as I thought since certbot has a post-create/update/delete command that you can add to run automatically when it does it's stuff. I'll do a few tests today at least then, once all the symlinks are out of the way, it will be clear if it's an nginx issue. – vy218 Jun 20 '19 at 08:48
As default certbot is installed and run under root so running it as non root user was first thing what I did :-). Until the location /etc/letsencrypt is writable for the user running certbot with first run it is ok. Once it is run as root for the first time the created structure should be re-owned as a easiest way. For the schedule I have utilized cron so certbot is run as user certbot (created for that purpose) and in case of new cert the rest of stuff (incl. copying the cert to needed location) is done by cron under root... But as I wrote it is just one of the option ;-). – Kamil J Jun 20 '19 at 09:33

nginx config using variable in ssl_certificate path throws permissions error

1 Answers1

prepare instead of linking

run certbot as www-data user

Linked