MongoDB replicaset external access - keep getting internal cluster names

Question

I must be doing something terribly wrong. I have a replicaset configured using the MongoDB community operator, deployed in GKE, and exposed via LoadBalancers.

This replicaset has 3 members. I have defined the replicaSetHorizons like so:

  replicaSetHorizons:
  - mongo-replica: document-0.mydomain.com:30000
  - mongo-replica: document-1.mydomain.com:30001
  - mongo-replica: document-2.mydomain.com:30002

I then use mongosh from an external source (local computer outside of GKE) to connect:

mongosh "mongodb://<credentials>@document-0.mydomain.com:30000,document-1.mydomain.com:30001,document-2.mydomain.com:30002/admin?ssl=false&replicaSet=document"

I do not use SSL for now because I am testing this deployment. What I found is mongosh always returns this error:

MongoNetworkError: getaddrinfo ENOTFOUND document-0.document-svc.mongodb.svc.cluster.local

Can someone explain to me what I am doing wrong? Why is my internal clustername being given to mongosh to attempt the connection?

If I try to connect to a single member of the replicaset, the connection will succeed. If I run rs.conf(), I see the following (which looks correct??):

{
  _id: 'document',
  version: 1,
  term: 1,
  members: [
    {
      _id: 0,
      host: 'document-0.document-svc.mongodb.svc.cluster.local:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      horizons: { 'mongo-replica': 'document-0.mydomain.com:30000' },
      secondaryDelaySecs: Long("0"),
      votes: 1
    },
    {
      _id: 1,
      host: 'document-1.document-svc.mongodb.svc.cluster.local:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      horizons: { 'mongo-replica': 'document-1.mydomain.com:30001' },
      secondaryDelaySecs: Long("0"),
      votes: 1
    },
    {
      _id: 2,
      host: 'document-2.document-svc.mongodb.svc.cluster.local:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      horizons: { 'mongo-replica': 'document-2.mydomain.com:30002' },
      secondaryDelaySecs: Long("0"),
      votes: 1
    }
  ],
  protocolVersion: Long("1"),
  writeConcernMajorityJournalDefault: true,
  settings: {
    chainingAllowed: true,
    heartbeatIntervalMillis: 2000,
    heartbeatTimeoutSecs: 10,
    electionTimeoutMillis: 10000,
    catchUpTimeoutMillis: -1,
    catchUpTakeoverDelayMillis: 30000,
    getLastErrorModes: {},
    getLastErrorDefaults: { w: 1, wtimeout: 0 },
    replicaSetId: ObjectId("62209784e8aacd8385db1609")
  }
}

My apologies. I have edited the question. Thank you for pointing that out. — jsy, Mar 03 '22 at 22:44

score 1 · Accepted Answer · answered Apr 11 '22 at 14:01

ReplicaSetHorizons feature does not work without using SSL/TLS certificates. Quoting from Kubernetes Operator reference:

This method to use split horizons requires the Server Name Indication extension of the TLS protocol

In order to make this work, you need to include

TLS certificate
TLS key
CA key

TLS Certificate must contain DNS names of all your replica sets in Subject Alternative Name (SAN) section

There is a tutorial at operator github pages. You need to complete all steps, certificate issuance cannot be skipped.

Certificate resource (using cert-manager.io CRD)

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cert-manager-certificate
spec:
  secretName: mongodb-tls
  issuerRef:
    name: ca-issuer
    kind: Issuer
  duration: 87600h
  commonName: "*.document-svc.mongodb.svc.cluster.local"
  dnsNames:
    - "*.document-svc.mongodb.svc.cluster.local"
    - "document-0.mydomain.com"
    - "document-1.mydomain.com"
    - "document-2.mydomain.com"

MongoDBCommunity resource excerpt

spec:
  type: ReplicaSet
  ...
  replicaSetHorizons:
    - mongo-replica: document-0.mydomain.com:30000
    - mongo-replica: document-0.mydomain.com:30001
    - mongo-replica: document-0.mydomain.com:30002
  security:
    tls:
      enabled: true
      certificateKeySecretRef:
        name: mongodb-tls
      caConfigMapRef:
        name: ca-config-map

Secret mongodb-tls will by of type tls and contain ca.crt, tls.crt and tls.key fields representing Certificate Authority certificate, TLS certificate and TLS key respectively.
ConfigMap ca-config-map will contain ca.crt field only

More info at: mongodb-operator-secure-tls

I take it only an internal CA can be used to generate this type of cert? I can't use a public CA because no public CA will verify private DNS namespaces. This means a Kubernetes deployment of MongoDB can never use a public cert? And a public cert protected MongoDB instance can only work with say, EC2 instances? — jsy, Apr 12 '22 at 07:27
I assume you are right about public CA not providing certificate for internal DNS namespaces (never tried it though). Therefore it is impossible to use public cert. Deployment on EC2 is not deployment on Kubernetes, but yes, you can make it work with public certs there. I ended up deploying both database and its clients in single cluster. — Peter Podhorsky, Apr 19 '22 at 19:38

MongoDB replicaset external access - keep getting internal cluster names

1 Answers1