4

I have setup the mongodb replicaset on kubernetes with Statefulsets using this helm chart

I can access the mongo instance inside the cluster. But I'd like to open it for access to the external world. I tried two approaches to accomplish it

  1. Create an additional service of type 'NodePort' mapping to the mongo instances with the selector label.

  2. Expose all 3 mongodb pods externally.

    kubectl expose pods mongo-release-mongodb-replicaset-2 --type=NodePort

Here is my test script. 

 from pymongo import MongoClient
 client = MongoClient('192.168.99.100',30738) #approach 1
 #client = MongoClient('mongodb://192.168.99.100:31455,192.168.99.100:31424,192.168.99.100:31569/?replicaSet=rs0') #approach 2
 db=client.test
 db.test.insert({"key1":1})
 values=db.test.find({'key1': 1})
 for value in values:
    print value

With the first approach, I get the following error which makes sense, since the external service is not consistently hitting the primary node in replicaset. Multiple attempts would eventually connect to the master and the write works.

File "/Library/Python/2.7/site-packages/pymongo/pool.py", line 552, in _raise_connection_failure
raise error
pymongo.errors.NotMasterError: not master

With the second approach, since we are accessing each pod directly by their IP:port, I expected it to work but it throws the following exception

pymongo.errors.ServerSelectionTimeoutError: mongo-release-mongodb-replicaset-0.mongo-release-mongodb-replicaset.default.svc.cluster.local:27017: [Errno 8] nodename nor servname provided, or not known,mongo-release-mongodb-replicaset-2.mongo-release-mongodb-replicaset.default.svc.cluster.local:27017: [Errno 8] nodename nor servname provided, or not known,mongo-release-mongodb-replicaset-1.mongo-release-mongodb-replicaset.default.svc.cluster.local:27017: [Errno 8] nodename nor servname provided, or not known

From the error it appears that the DNS translation is causing issues? I looked at this question but didn't get much help out of it

I'm running out of ideas. Can anyone help with this issue? Any alternative solutions are appreciated as well.

Thanks

Srikanth
  • 125
  • 2
  • 11

2 Answers2

2

After spending some more time with the issue, I figured that the mongo endpoint in the above script was returning the replicaset's DNS' in the following format - 'mongo-release-mongodb-replicaset-0.mongo-release-mongodb-replicaset.default.svc.cluster.local:27017'. These addresses can only be resolved within the cluster namespace. Verified the same by running the following script in another pod 

from pymongo import MongoClient
client = MongoClient('mongodb://mongo-release-mongodb-replicaset-0.mongo-release-mongodb-replicaset:27017,mongo-release-mongodb-replicaset-1.mongo-release-mongodb-replicaset:27017,mongo-release-mongodb-replicaset-2.mongo-release-mongodb-replicaset:27017/?replicaSet=rs0')
db=client.test
db.test.insert({"key1":1})
values=db.test.find({'key1': 1})
for value in values:
   print value
Srikanth
  • 125
  • 2
  • 11
  • Indeed, if you create a Service of type NodePort you have to hit the IP of the nodes that expose the port (private/public network interface). The service name is resolvable only by kube-dns, and the returned IP is internal only (pods/cluster nodes). – Antoine Cotten Jun 25 '17 at 23:21
  • The major issue is that MongoDB clusters are providing their topology to the client based on the configuration you provided to initiate the replicaset. The external endpoint is aonly used to get this topology and then the client tries to directly access the mongo nodes. You need multiple points: - the names provided in the rs.initiate() must be resolvable from your client - the resolved IP must be routable from your client – vgkowski Nov 02 '17 at 15:27
1

A workaround for this problem is to instantiate the MongoClient with the parameter directConnection=True, like this:

c = MongoClient('localhost', 27017, directConnection=True)

PyMongo docs: https://pymongo.readthedocs.io/en/stable/examples/high_availability.html?highlight=dns

Mauri Francisco do Carmo
  • 11
  • 1