MongoDB Kubernetes connection failure with SSL

Question

I have three node MongoDB cluster in GCP kubernetes cluster following [1], [2]. I can properly connect with tls=false using mongosh client. Then I enabled tls following [3]. Mongo cluster start properly but I cannot connect from mongosh.

Following is the connection details.

{
  "connectionString.standard": "mongodb://mongo-user:stl-m0ng0-dev@mongodb-dev-0.mongodb-dev-svc.dev.svc.cluster.local:27017,mongodb-dev-1.mongodb-dev-svc.dev.svc.cluster.local:27017,mongodb-dev-2.mongodb-dev-svc.dev.svc.cluster.local:27017/dev?replicaSet=mongodb-dev&ssl=true",
  "connectionString.standardSrv": "mongodb+srv://mongo-user:stl-m0ng0-dev@mongodb-dev-svc.dev.svc.cluster.local/dev?replicaSet=mongodb-dev&ssl=true",
  "password": "xxxxxxx",
  "username": "mongo-user"
}

Followings are the certificate details.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2 (0x2)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = TLSGenSelfSignedtRootCA, L = $$$$
        Validity
            Not Before: Jul 27 09:07:50 2022 GMT
            Not After : Jul 24 09:07:50 2032 GMT
        Subject: CN = *.mongodb-dev-svc.dev.svc.cluster.local, O = client
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:c4:44:a6:21:95:85:9a:dc:96:63:8e:76:ed:d9:
                    3a:59
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
            X509v3 Subject Alternative Name:
                DNS:mongodb-dev-1.mongodb-dev-svc.dev.svc.cluster.local, DNS:mongodb-dev-2.mongodb-dev-svc.dev.svc.cluster.local, DNS:mongodb-dev-3.mongodb-dev-svc.dev.svc.cluster.local
    Signature Algorithm: sha256WithRSAEncryption
         7b:78:43:73:ae:2f:ce:97:de:b2:19:56:4c:38:71:8e:3d:ff:
         5b:15:79:c1
Will display server certificate info


Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = TLSGenSelfSignedtRootCA, L = $$$$
        Validity
            Not Before: Jul 27 09:07:50 2022 GMT
            Not After : Jul 24 09:07:50 2032 GMT
        Subject: CN = *.mongodb-dev-svc.dev.svc.cluster.local, O = server
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:bc:1e:4a:a7:4f:c4:01:71:2c:78:eb:ac:c9:53:
                    24:c1
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                DNS:mongodb-dev-0.mongodb-dev-svc.dev.svc.cluster.local, DNS:mongodb-dev-1.mongodb-dev-svc.dev.svc.cluster.local, DNS:mongodb-dev-2.mongodb-dev-svc.dev.svc.cluster.local
    Signature Algorithm: sha256WithRSAEncryption
         16:0f:09:02:66:05:69:7b:91:3b:93:73:86:64:d5:8f:53:2d:
         08:19:68:a7

Client side has following error

root@xxxxxxxxxxxxxxxxxx-55955c9fcd-bpp98:/usr/src/app# mongosh "mongodb+srv://mongo-user:stl-m0ng0-dev@mongodb-dev-svc.dev.svc.cluster.local/dev?replicaSet=mongodb-dev&ssl=false&tlsCAFile=ca.pem&tlsCertificateKeyFile=key.pem"                                                                                                            
Current Mongosh Log ID: 62e1029487b960f1bd204b1d
Connecting to:          mongodb+srv://<credentials>@mongodb-dev-svc.dev.svc.cluster.local/dev?replicaSet=mongodb-dev&ssl=false&tlsCAFile=ca.pem&tlsCertificateKeyFile=key.pem&appName=mongosh+1.5.1
MongoServerSelectionError: connection <monitor> to 10.120.6.8:27017 closed

Server side has following error

2022-07-27T09:25:44.992+0000 I  NETWORK  [conn25852] end connection 10.120.6.9:33914 (14 connections now open)
2022-07-27T09:25:44.993+0000 I  NETWORK  [listener] connection accepted from 10.120.6.9:33918 #25855 (15 connections now open)
2022-07-27T09:25:44.993+0000 E  NETWORK  [conn25854] SSL peer certificate validation failed: unsupported certificate purpose
2022-07-27T09:25:44.994+0000 I  NETWORK  [conn25854] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unsupported certificate purpose. Ending connection from 10.120.8.127:58220 (connection id: 25854)
2022-07-27T09:25:44.994+0000 I  NETWORK  [conn25854] end connection 10.120.8.127:58220 (14 connections now open)
2022-07-27T09:25:44.995+0000 I  NETWORK  [listener] connection accepted from 10.120.8.127:58224 #25856 (15 connections now open)
2022-07-27T09:25:44.998+0000 E  NETWORK  [conn25855] SSL peer certificate validation failed: unsupported certificate purpose
2022-07-27T09:25:44.998+0000 I  NETWORK  [conn25855] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unsupported certificate purpose. Ending connection from 10.120.6.9:33918 (connection id: 25855)
2022-07-27T09:25:44.998+0000 I  NETWORK  [conn25855] end connection 10.120.6.9:33918 (14 connections now open)
2022-07-27T09:25:45.000+0000 E  NETWORK  [conn25856] SSL peer certificate validation failed: unsupported certificate purpose
2022-07-27T09:25:45.000+0000 I  NETWORK  [conn25856] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unsupported certificate purpose. Ending connection from 10.120.8.127:58224 (connection id: 25856)
2022-07-27T09:25:45.000+0000 I  NETWORK  [conn25856] end connection 10.120.8.127:58224 (13 connections now open)
2022-07-27T09:25:45.001+0000 I  REPL_HB  [replexec-2] Heartbeat to mongodb-dev-1.mongodb-dev-svc.dev.svc.cluster.local:27017 failed after 2 retries, response status: HostUnreachable: stream truncated
2022-07-27T09:25:45.003+0000 I  NETWORK  [listener] connection accepted from 10.120.8.127:58228 #25858 (14 connections now open)
2022-07-27T09:25:45.007+0000 E  NETWORK  [conn25858] SSL peer certificate validation failed: unsupported certificate purpose
2022-07-27T09:25:45.007+0000 I  NETWORK  [conn25858] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unsupported certificate purpose. Ending connection from 10.120.8.127:58228 (connection id: 25858)
2022-07-27T09:25:45.007+0000 I  NETWORK  [conn25858] end connection 10.120.8.127:58228 (13 connections now open)

Operator log has TLS configuration issue.

2022-07-27T10:06:05.893Z        INFO    controllers/mongodb_status_options.go:110       TLS config is not yet valid, retrying in 10 seconds
2022-07-27T10:06:15.899Z        INFO    controllers/replica_set_controller.go:140       Reconciling MongoDB     {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z        DEBUG   controllers/replica_set_controller.go:142       Validating MongoDB.Spec {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z        DEBUG   controllers/replica_set_controller.go:151       Ensuring the service exists     {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z        DEBUG   agent/agent_readiness.go:101    The Pod '' doesn't have annotation 'agent.mongodb.com/version' yet      {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z        DEBUG   agent/agent_readiness.go:101    The Pod '' doesn't have annotation 'agent.mongodb.com/version' yet      {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z        DEBUG   agent/agent_readiness.go:101    The Pod '' doesn't have annotation 'agent.mongodb.com/version' yet      {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.906Z        INFO    controllers/replica_set_controller.go:462       Create/Update operation succeeded       {"ReplicaSet": "dev/mongodb-replica-set","operation": "updated"}
2022-07-27T10:06:15.906Z        INFO    controllers/mongodb_tls.go:40   Ensuring TLS is correctly configured    {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.906Z        WARN    controllers/mongodb_tls.go:47   CA resource not found: Secret "tls-ca-key-pair" not found       {"ReplicaSet": "dev/mongodb-replica-set"}

[1]. https://github.com/mongodb/mongodb-kubernetes-operator/blob/v0.7.4/docs/install-upgrade.md

[2]. https://github.com/mongodb/mongodb-kubernetes-operator/blob/v0.7.4/docs/deploy-configure.md

[3]. https://github.com/mongodb/mongodb-kubernetes-operator/blob/v0.7.4/docs/secure.md

Nuwan Sameera · Answer 1 · 2022-07-28T12:52:23.847

It had two main reasons.

I followed [1] to enable SSL. It create another Statefulset. After that there are two mongo servers. Uninstall operator and re-install and followed last stable release documentation [2]. After it properly detect configmap and secret.

But it gave SSL issue in certificates as Unsupported Certificate in server modules. Following [3] found the issue. We need to remove extended_key_useage from openssl.conf. Otherwise it not work properly.

Important thread [4]

Hope this help.

[1]. https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/docs/secure.md

[2]. https://github.com/mongodb/mongodb-kubernetes-operator/blob/v0.7.4/docs/secure.md

[3]. https://stackoverflow.com/a/61964464/5607943

[4]. https://groups.google.com/g/mongodb-user/c/EmESxx5KK9Q/m/xH6Ul7fTBQAJ

MongoDB Kubernetes connection failure with SSL

1 Answers1