Trusting self signed certificate on Amazon Linux 2

Question

I'm aware this question has been asked like a million times but no answer or comment has helped me so far. I'm running in circles since a full week without any progress.

Situation:

I am on an Amazon Linux 2 system
I have a self-signed certificate (this is generted by iamlive - iamlive is running as a proxy and inspects all traffic to AWS endpoints to record IAM polciies)

The iamlive command looks like this:

iamlive --bind-addr "0.0.0.0:12345" --mode proxy --ca-bundle /etc/pki/ca-trust/source/anchors/iamlive.pem --ca-key /etc/pki/tls/private/iamlive.key

When the cert and key files do not exist, they will be created. When they do exist, they will be used.
After the certificate has been created I run update-ca-trust. I confirmed, then the file /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt includes the iamlive certificate.

Now I would expect that the whole system (except applications which hold their own certificate store) will trust this certificate.

But when I run, for example, the aws cli it will complain about a self signed certificate:

$ export HTTPS_PROXY=http://127.0.0.1:12345
$ export HTTP_PROXY=http://127.0.0.1:12345
$ aws s3 ls

SSL validation failed for https://s3.eu-central-1.amazonaws.com/ [SSL: CERTIFICATE_VERIFY_FAILED] 
certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)

Of course I am aware I can tell the cli to use a specific CA bundle and that perfectly works:

$ export HTTPS_PROXY=http://127.0.0.1:12345
$ export HTTP_PROXY=http://127.0.0.1:12345
$ export AWS_CA_BUNDLE=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
$ aws s3 ls

... listing all the buckets

But here is the thing I don't get. From my understanding, the file /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt is what openssl should be using by default anyway. So why would it work only when I explicitly set the default file?

Now with the aws cli I can work with the env var AWS_CA_BUNDLE, which is fine.

Alternatively, there is REQUESTS_CA_BUNDLE, which does the same job, for everything that uses the python requests package, which is used by boto3 (which is used by the aws cli).

My problem is, that I need this to work with Ansible, which also uses botocore. And I have not found a way to set the CA bundle path for that, since botocore does not use the requests package.

When I run ansible-playbook (with AWS_CA_BUNDLE and REQUESTS_CA_BUNDLE both pointing to /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt) which execute any task that connects over http(s), I will get an error like this:

Traceback (most recent call last):
  File \"/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 700, in urlopen
    self._prepare_proxy(conn)
  File \"/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 996, in _prepare_proxy
    conn.connect()
  File \"/usr/local/lib/python3.7/site-packages/urllib3/connection.py\", line 424, in connect
    tls_in_tls=tls_in_tls,
  File \"/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py\", line 450, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File \"/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py\", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File \"/usr/lib64/python3.7/ssl.py\", line 423, in wrap_socket
    session=session
  File \"/usr/lib64/python3.7/ssl.py\", line 870, in _create
    self.do_handshake()
  File \"/usr/lib64/python3.7/ssl.py\", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/usr/local/lib/python3.7/site-packages/botocore/httpsession.py\", line 464, in send
    chunked=self._chunked(request.headers),
  File \"/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 788, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File \"/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py\", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File \"/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py\", line 769, in reraise
    raise value.with_traceback(tb)
  File \"/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 700, in urlopen
    self._prepare_proxy(conn)
  File \"/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 996, in _prepare_proxy
    conn.connect()
  File \"/usr/local/lib/python3.7/site-packages/urllib3/connection.py\", line 424, in connect
    tls_in_tls=tls_in_tls,
  File \"/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py\", line 450, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File \"/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py\", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File \"/usr/lib64/python3.7/ssl.py\", line 423, in wrap_socket
    session=session
  File \"/usr/lib64/python3.7/ssl.py\", line 870, in _create
    self.do_handshake()
  File \"/usr/lib64/python3.7/ssl.py\", line 1139, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/home/user/.ansible/tmp/ansible-tmp-1666344891.9240296-34-66835764316077/AnsiballZ_ec2_instance_info.py\", line 102, in <module>
    _ansiballz_main()
  File \"/home/user/.ansible/tmp/ansible-tmp-1666344891.9240296-34-66835764316077/AnsiballZ_ec2_instance_info.py\", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File \"/home/user/.ansible/tmp/ansible-tmp-1666344891.9240296-34-66835764316077/AnsiballZ_ec2_instance_info.py\", line 40, in invoke_module
    runpy.run_module(mod_name='ansible.modules.cloud.amazon.ec2_instance_info', init_globals=None, run_name='__main__', alter_sys=True)
  File \"/usr/lib64/python3.7/runpy.py\", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File \"/usr/lib64/python3.7/runpy.py\", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File \"/usr/lib64/python3.7/runpy.py\", line 85, in _run_code
    exec(code, run_globals)
  File \"/tmp/ansible_ec2_instance_info_payload_r6ce4qil/ansible_ec2_instance_info_payload.zip/ansible/modules/cloud/amazon/ec2_instance_info.py\", line 564, in <module>
  File \"/tmp/ansible_ec2_instance_info_payload_r6ce4qil/ansible_ec2_instance_info_payload.zip/ansible/modules/cloud/amazon/ec2_instance_info.py\", line 560, in main
  File \"/tmp/ansible_ec2_instance_info_payload_r6ce4qil/ansible_ec2_instance_info_payload.zip/ansible/modules/cloud/amazon/ec2_instance_info.py\", line 512, in list_ec2_instances
  File \"/usr/local/lib/python3.7/site-packages/botocore/paginate.py\", line 479, in build_full_result
    for response in self:
  File \"/usr/local/lib/python3.7/site-packages/botocore/paginate.py\", line 269, in __iter__
    response = self._make_request(current_kwargs)
  File \"/usr/local/lib/python3.7/site-packages/botocore/paginate.py\", line 357, in _make_request
    return self._method(**current_kwargs)
  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 514, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 922, in _make_api_call
    operation_model, request_dict, request_context
  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 944, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File \"/usr/local/lib/python3.7/site-packages/botocore/endpoint.py\", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File \"/usr/local/lib/python3.7/site-packages/botocore/endpoint.py\", line 207, in _send_request
    exception,
  File \"/usr/local/lib/python3.7/site-packages/botocore/endpoint.py\", line 361, in _needs_retry
    request_dict=request_dict,
  File \"/usr/local/lib/python3.7/site-packages/botocore/hooks.py\", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File \"/usr/local/lib/python3.7/site-packages/botocore/hooks.py\", line 256, in emit
    return self._emit(event_name, kwargs)
  File \"/usr/local/lib/python3.7/site-packages/botocore/hooks.py\", line 239, in _emit
    response = handler(**kwargs)
  File \"/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py\", line 207, in __call__
    if self._checker(**checker_kwargs):
  File \"/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py\", line 285, in __call__
    attempt_number, response, caught_exception
  File \"/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py\", line 320, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File \"/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py\", line 364, in __call__
    attempt_number, response, caught_exception
  File \"/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py\", line 248, in __call__
    attempt_number, caught_exception
  File \"/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py\", line 416, in _check_caught_exception
    raise caught_exception
  File \"/usr/local/lib/python3.7/site-packages/botocore/endpoint.py\", line 281, in _do_get_response
    http_response = self._send(request)
  File \"/usr/local/lib/python3.7/site-packages/botocore/endpoint.py\", line 377, in _send
    return self.http_session.send(request)
  File \"/usr/local/lib/python3.7/site-packages/botocore/httpsession.py\", line 482, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://ec2.eu-central-1.amazonaws.com/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)

Now even if there was a magic env var I could set to get this to work for Ansible, it would't be a complete solution. The same problem persists with the AWS Data Provider for SAP and probably a million more tools I haven't even tested yet.

So the question is: How to trust this self-signed cert globally, so that all apps accept it?

Apparently you're right. I just took a journey into the python code of botocore/urllib3 and found this is using the python package certifi. Hacked into that and found this is using the hardcoded file `/usr/local/lib/python3.7/site-packages/certifi/cacert.pem`, so it ships with the package. I was able to get Ansible working by deleting that file and symlinking my ca-bundle. Just wondering what the file `/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt` is actually good for when you need to force every single app in one way or another to use this. — udondan, Oct 21 '22 at 13:17

Trusting self signed certificate on Amazon Linux 2

0 Answers0