3

From last 2 days we started seeing chef execution failures are happening related to apt_repository resource. Seen similar failures with chef-client execution too

Chef-solo version: 12.19.36 OS: Ubuntu v18 & Ubuntuv14

See below error from chef-solo execution.

==> core: [2021-10-04T14:36:46+00:00] ERROR: SSL Validation failure connecting to host: www.postgresql.org - SSL_connect returned=1 errno=0 state=error: certificate verify failed
==> core:
==> core:
==> core: ================================================================================
==> core: Error executing action create on resource 'remote_file[/var/chef/cache/https___www_postgresql_org_media_keys_ACCC4CF8_asc]'
==> core: ================================================================================
==> core:
==> core: OpenSSL::SSL::SSLError
==> core: ----------------------
==> core: SSL Error connecting to https://www.postgresql.org/media/keys/ACCC4CF8.asc - SSL_connect returned=1 errno=0 state=error: certificate verify failed
==> core:
==> core:
==> core: Resource Declaration:
==> core: ---------------------
==> core: # In /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/provider/apt_repository.rb
==> core:
==> core: 166: declare_resource(type, cached_keyfile) do
==> core: 167: source new_resource.key
==> core: 168: mode "0644"
==> core: 169: sensitive new_resource.sensitive
==> core: 170: action :create
==> core: 171: end
==> core: 172:
==> core:
==> core: Compiled Resource:
==> core:
==> core: ------------------
==> core: # Declared in /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/provider/apt_repository.rb:166:in `install_key_from_uri'
==> core:
Parepallykiran
  • 41
  • 3

3 Answers3

4

This is related to last Thursday's Root CA Certificate Expiration for Let's Encrypt certificates.

For OpenSSL, you need to upgrade to at least 1.1 I believe due to a bug with how 1.0.1 handles the certificate chain. If that's not possible you will need to remove the DST Root CA X3 certificate from the OS.

Chef also provides it's own root certificates it uses instead of the OS. I had to remove the above certificate from /opt/chef/embedded/ssl/certs/cacert.pem and also add ISRG Root X1 to it.

Dave Rager
  • 8,002
  • 3
  • 33
  • 52
  • 1
    Thanks Dave for the response. Issue is resolved by following your recommended steps. – Parepallykiran Oct 04 '21 at 19:53
  • Any chance you remember the commands to remove the cert from /opt/chef/embedded/ssl/certs/cacert.pem? – kivagant Oct 29 '21 at 03:17
  • 1
    @kivagant My workaround (for now) involved making a copy of that pem, editing it with a text editor to make the above changes and created a chef recipe that replaces the original with the updated version. This recipe is the first in the run list so the change is made before any other recipes run. – Dave Rager Oct 29 '21 at 14:34
  • I hoped there's a way to avoid fully copying it to a recipe templates. I tried to replace completely with the chain from the latest `ca-certificates`, but it causes unclear issues in AWS OpsWorks. Thank you for sharing your experience. – kivagant Oct 29 '21 at 16:44
  • Facing the same while running `chef-client`. Is it enough to upgrade openssl to version 1.1 at the node or should it be upgraded at both the node and the host? – Shriram Balakrishnan Dec 04 '21 at 13:49
  • @DaveRager - As per your answer, I upgraded to openssl `1.1.1l` version. But I still faced the same issue. But removing `DST Root CA X3` from `/opt/chef/embedded/ssl/certs/cacert.pem` made `chef-client` run successfully. Any idea why the openssl upgrade didn't make this work? – Shriram Balakrishnan Dec 04 '21 at 16:28
  • @ShriramBalakrishnan I do not know why the upgrade did not work. I did not try it myself. According to Let's Encrpyt's announcement it should have worked. https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ – Dave Rager Dec 04 '21 at 23:28
0

The simpler fix, which also keeps your Chef client certificates in sync with your system, is to allow Chef client to use your system certificates - assuming you are not getting these errors outside of the Chef client runs.

To do this, remove the existing symbolic link, then re-create it pointing to your system certificates (Ubuntu 18.04 in this example):

$ ls -l /opt/chefdk/embedded/ssl/cert.pem
lrwxrwxrwx 1 root root 16 Jun  2  2020 /opt/chefdk/embedded/ssl/cert.pem -> certs/cacert.pem

# sudo rm /opt/chefdk/embedded/ssl/cert.pem
# sudo ln -s /etc/ssl/certs/ca-certificates.crt /opt/chefdk/embedded/ssl/cert.pem

$ ls -l /opt/chefdk/embedded/ssl/cert.pem
lrwxrwxrwx 1 root root 34 Mar 10 09:16 /opt/chefdk/embedded/ssl/cert.pem -> /etc/ssl/certs/ca-certificates.crt

To check that all is OK:

$ /opt/chefdk/embedded/bin/openssl s_client -connect some.domain.com:443 -showcerts
CONNECTED(00000003)                                             
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1         
verify return:1                                                 
depth=1 C = US, O = Let's Encrypt, CN = R3                      
verify return:1                                                                 
depth=0 CN = some.domain.com                                    
verify return:1
---                                                             
Certificate chain 

<snip>

    Start Time: 1646864178                                      
    Timeout   : 300 (sec)                                       
    Verify return code: 0 (ok)

<snip>
Hedgehog
  • 5,487
  • 4
  • 36
  • 43
-1

You're on a pretty old version of Chef there. Note that chef has since released new versions that remove the expired certs as well as upgrading OpenSSL. The latest Chef 16 & 17 versions should address this problem.

Don Seiler
  • 460
  • 5
  • 16