Apache HTTPD occasionally serving old/expired certificates

Question

I have three hosts on the same server (single IP):

domain1.com
domain2.com
domain3.com

Each of them is supposed to use a Let's Encrypt certificate, recently issued in April 2017.

However, occasionally the server seems to serve old (expired) certificates. In case of domain1.com, an actually still valid StartSSL certificate (07.2016 - 07-2017) is served, while the other two hosts serve expired Let's Encrypt certificates from January 2017.

Here is how each of these hosts is basically set up (with different host names, of course) for certbot:

<VirtualHost *:443>

    ServerName  domain1.com
    ServerAlias www.domain1.com

    SSLEngine           on
    SSLProtocol         all -SSLv2 -SSLv3
    SSLCipherSuite      ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
    SSLHonorCipherOrder on
    SSLCompression      off

    SSLOptions +StrictRequire

    SSLCertificateFile /etc/letsencrypt/live/domain1.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/domain1.com/privkey.pem

    DocumentRoot /opt/lucee/tomcat/webapps/domain1.com/

    <IfModule mod_headers.c>
        RequestHeader set HTTPS "1"
    </IfModule>
    <IfModule mod_proxy.c>
        ProxyPassMatch ^/(.*)$ http://127.0.0.1:8500/$1
    </IfModule>

    CustomLog ${APACHE_LOG_DIR}/access.log custom_access

</VirtualHost>

fullchain.pem and privkey.pem are symlinks and point to the most recent files (highest index) in the archive folder for each host. I resolved the links and it seems fine.

#apache2ctl -S

VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server localhost (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost localhost (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost domain1.com (/etc/apache2/sites-enabled/000-default.conf:8)
                 alias www.domain1.com
         port 80 namevhost domain2.com (/etc/apache2/sites-enabled/000-default.conf:17)
                 alias www.domain2.com
         port 80 namevhost domain3.com (/etc/apache2/sites-enabled/000-default.conf:26)
                 alias www.domain3.com
         port 80 namevhost www.domain2.com (/etc/apache2/sites-enabled/000-default.conf:35)
                 alias domain2.com
         port 80 namevhost forum.domain2.com (/etc/apache2/sites-enabled/000-default.conf:44)
         port 80 namevhost downloads.domain2.com (/etc/apache2/sites-enabled/000-default.conf:69)
         port 80 namevhost images.domain2.com (/etc/apache2/sites-enabled/000-default.conf:82)
*:443                  is a NameVirtualHost
         default server domain1.com (/etc/apache2/sites-enabled/001-domain1.com.conf:3)
         port 443 namevhost domain1.com (/etc/apache2/sites-enabled/001-domain1.com.conf:3)
                 alias www.domain1.com
         port 443 namevhost www.domain2.com (/etc/apache2/sites-enabled/002-www.domain2.com.conf:3)
                 alias domain2.com
         port 443 namevhost domain3.com (/etc/apache2/sites-enabled/003-domain3.com.conf:3)
                 alias www.domain3.com
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex default: dir="/var/lock/apache2" mechanism=fcntl
Mutex mpm-accept: using_defaults
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33

#certbot certificates

-------------------------------------------------------------------------------
Found the following certs:
  Certificate Name: www.domain2.com
    Domains: www.domain2.com
    Expiry Date: 2017-07-02 23:03:00+00:00 (VALID: 75 days)
    Certificate Path: /etc/letsencrypt/live/www.domain2.com/fullchain.pe                                                  m
    Private Key Path: /etc/letsencrypt/live/www.domain2.com/privkey.pem
  Certificate Name: domain3.com
    Domains: domain3.com
    Expiry Date: 2017-07-02 23:01:00+00:00 (VALID: 75 days)
    Certificate Path: /etc/letsencrypt/live/domain3.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/domain3.com/privkey.pem
  Certificate Name: domain1.com
    Domains: domain1.com
    Expiry Date: 2017-07-02 23:03:00+00:00 (VALID: 75 days)
    Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem
-------------------------------------------------------------------------------

What could be the issue here? I always assumed the server returns multiple certificates due to SNI, but why does Apache HTTPD mix up different certificates? Where do these even come from? (And yes, I restarted and reloaded Apache several times.)

What do you mean by "occassionally"? This rather implies that sometimes it serves a new certificate for a specific site, sometimes an old certificate. Have you checked what files are on the server? Did you check this is configured for SNI (since you seem to be using the same IP address for all hosts)? — symcbean, Apr 18 '17 at 16:56
I can request the domain and it serves the new cert. 20 seconds later it starts serving the old cert, rinse and repeat. The certs match the host at all times though. The old certs are present in the `archive` folder, but they are not attached to the Apache vHosts (unless symlinks keep changing their destination). I don't think there's any option to configure SNI as Apache does it automatically when there are more than one vHost listening on 443. ssllabs.com tells me that the certs are fine, but browsers keep telling me that the cert expired and some seconds later it's fine again. — Alex, Apr 18 '17 at 18:45
You might set a header that gives a unique identifier of the web server behind your load balancer, so you can see which one has the stale data. — Aaron, Apr 18 '17 at 19:16

score 3 · Accepted Answer · answered Apr 19 '17 at 05:32

3

I found the reason. I had multiple instances of Apache running. Restarting/Reloading via terminal only restarted some of them. That explains why sometimes the old certs and sometimes the new certs were served. I had to kill all processes to pick up the new certs and it's now working again.

answered Apr 19 '17 at 05:32

Alex

141
7

I experienced exactly the same issue, I thought the apache2 process would be killed after an sudo service apache2 restart, but no, you have to kill apache2 or better restart the server. Thank you very much @Alex. – leonardorame May 04 '21 at 16:19

Apache HTTPD occasionally serving old/expired certificates

1 Answers1