I am receiving SSL Certificate Verification Error from a gitlab CI build that uses docker in docker within a Python 3.x architecture. All CI build images are alpine linux based.
I have setup a CI Python 3.x test job that uses Pytest. Pytest spawns a docker-compose process in some of the tests. An image of the architecture can be found at the gitlab-ci forum post
From within the gitlab-CI test job I am exporting the REQUESTS_CA_BUNDLE and SSL_CERT_FILE environment variables to reference the custom root ca file. I have to set both variables since tus-py-client uses both the requests and http.client libraries for http requests.
The docker-compose service, started from pytest, encapsulates a custom ca root certificate. This is the server.
The custom ca root certificate is is also included within the image used by the gitlab CI build job that runs pytest. This is the client.
I have also tried encapsulating the ca root certificate in a custom docker:dind image. In all cases, update-ca-certificates was run when building the images. From what I understand, I think that the docker-compose container services will be started from within a docker-in-docker (dind) instance.
A brief overview of the .gitlab-ci.yml file is listed below.
image: <custom devops image encapulating root ca>
stages:
- test
variables:
DOCKER_DRIVER: overlay2
DOCKER_HOST: tcp://docker:2375
SHARED_PATH: ${CI_PROJECT_DIR}/fileserver
POSTGRES_DB: ${PG_DB}
POSTGRES_PASSWORD: ${PG_PASSWORD}
POSTGRES_USER: ${PG_USER}
DATABASE_URL: "postgres://${PG_USER}:${PG_PASSWORD}@postgres:5432/${PG_DB}"
services:
- name: <custom docker:dind image encapsulating ca root cert>
alias: docker
- name: <custom postgres image>
alias: postgres
before_script:
- <setup ssh key for cloning dependent repositories>
- <login to docker registry at registry.gitlab.com>
test:
stage: test
script:
- <clone and generate configuration for docker-compose project>
- pip install -r requirements.txt
- export REQUESTS_CA_BUNDLE=<ca root crt file>
- export SSL_CERT_FILE=<ca root crt file>
- pytest --cov=api --cov-report term-missing --cov-report html:coverage
From what I understand, the ca root certificate will need to be installed in the certificate store of both the client and server. In this particular case, the client is the CI test job and the server is an nginx container. This is triggered by the docker-compose process, spawned from within pytest.
An image of the architecture is available at my posting on the gitlab-ci forum.
I am getting a SSL Certificate Verificate failed exception, ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)? The client is connecting to the server in the docker:dind instance and the handshake process appears to be failing. Any ideas?
Update
If I issue openssl s_client -connect docker:1081 -CApath /etc/ssl/certs from within the CI build test job, to attempt verification of the certificate, I receive a verify error:num=21:unable to verify the first certificate and verify error:num=20:unable to get local issuer certificate.
I have tried starting up the docker-compose services on a separate Ubuntu 16.04 bare metal instance with the root certificate installed using update-ca-certificates. I then issued openssl s_client -connect docker:1081 -CApath /etc/ssl/certs which verified successfully after adding *docker 127.0.0.1 to /etc/hosts. So the server certificate and root ca appear to be ok.
Why am I receiving verify error numbers 20 and 21 when connecting to the specific docker-compose service running within docker:dind?
Update 21/12/2018 Fixed. In the image for the CI test job I placed the custom ca certificate in /usr/local/share/ca-certificates. Previously I placed the certificate in /usr/local/share/ca-certificates/subfolder and this was not getting appended to /etc/ssl/certs/ca-certificates.crt bundle after running update-ca-certificates. Following this, I reverted back to the de facto docker:dind image in .gitlab-ci.yml and the certificates were still verified successfully. So no need to install ca root cert in docker:dind.
Did a clean git clone in development environment. I fixed some issues with requirements.txt for pip install and successful tested in build environment. Build environment now correctly matching dev environment!
