Python client returns CERTIFICATE_VERIFY_FAILED on GRPC stub

Question

I have this easy code to connect to download some data using GRPC

creds = grpc.ssl_channel_credentials()
channel = grpc.secure_channel(f'{HOST}:{PORT}', credentials=creds)
stub = liveops_pb2_grpc.LiveOpsStub(channel=channel)
request = project_pb2.ListProjectsRequest(organization=ORGANIZATION)
projects = stub.ListProjects(request=request)
print(projects)

This worked fine on wednesday. It runs in a docker container with Python 3.8.10 and protobuf==3.18.0,grpcio==1.40.0, grpcio-tools==1.40.0.

Today I updated MAC OS Big Sur to 11.6 and after finishing some extra features on the code I see that it returns:

E0930 21:12:04.108551900       1 ssl_transport_security.cc:1468] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
E0930 21:12:04.194319000       1 ssl_transport_security.cc:1468] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
E0930 21:12:04.286163700       1 ssl_transport_security.cc:1468] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
Traceback (most recent call last):
  File "", line 302, in <module>
    projects = liveops_stub.ListProjects(request=request)
  File "/home/airflow/.local/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/airflow/.local/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "failed to connect to all addresses"
    debug_error_string = "{"created":"@1633036324.286560700","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3186,"referenced_errors":[{"created":"@1633036324.286548700","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":146,"grpc_status":14}]}"
>

Seems to be something related to SSL Certificates. If I check /etc/ssl/certs folder it is empty, so could be that SSL SO certificate has been erased?

How can I fix it?

score 1 · Answer 1 · answered Oct 01 '21 at 11:10

1

Let's Encrypt cross-signed DST Root CA X3 expired yesterday, it caused some SSL problems on some clients. I managed to fix it by manually changing the intermediate chain to use the new Root X1 instead. On the server, run:

sudo certbot certonly --nginx -d <ADDRESS> --preferred-chain "ISRG Root X1"

If you are not running nginx you might need to replace it with your server, and make sure to restart it.

answered Oct 01 '21 at 11:10

Firas Al Mannaa

916
1
11
30

Yes, i just fixed it. I will provide an answer too as I am not running nginx. Thanks! – mrc Oct 01 '21 at 11:35

score 1 · Answer 2 · answered Oct 01 '21 at 11:36

As Firas Al Mannaa said Let's Encrypt cross-signed DST Root CA X3 expired yesterday, it caused some SSL problems on some clients.

I added a new function to get a certificate from ca-bundle:

def get_ssl_certificate():
    request_response = req.get('https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt')
    certificate_content = request_response.text
    certificate_as_bytes = str.encode(certificate_content)

    return certificate_as_bytes

And then I used this certificate

creds = grpc.ssl_channel_credentials(root_certificates=certificate_as_bytes)

Python client returns CERTIFICATE_VERIFY_FAILED on GRPC stub

2 Answers2