1

I'm using Mongodb with only one shard and 3 members in a replica set. I have each replica set member on a different machine. Two members each have a mongos router, one member (that has a router) has a config server, and all three have a shard server. I have one hostname that resolves to the two ip addresses of each one of the mongodb instances that has a mongos router. I'm using mongoose v.2.5.10 to connect to mongodb from my node.js application with the following code:

var mongoose = require('mongoose');
mongoose.connection.on('error', function(err){
  console.error('Mongodb connection error: ', err);
});
var db = mongoose.connect('mongodb://username:password@' + mongoHost + '/database');

I simulate what would happen in a failover scenario by terminating one of the mongodb instances that has the mongos router, and when this happens, I detect that the instance is down and prune the instance's ip address from the DNS record. However, my application does not seem to correctly reconnect to mongodb, as the error event of the mongoose connection is not being emitted, and my application hangs until I restart my node server.

What might be going wrong here?

UPDATE:

I've just confirmed that the net client that the mongodb node.js client is using to connect (this.connection = net.createConnection(this.socketOptions.port, this.socketOptions.host);) is not receiving the close or error event when I manually terminate the Amazon instance. So this seems like this could be an internal node issue...

Justin Meltzer
  • 13,318
  • 32
  • 117
  • 182
  • See if my question helps you: http://stackoverflow.com/questions/16226472/mongoose-autoreconnect-option – gustavohenke Jun 09 '13 at 01:13
  • Ah, also, update your mongoose version, it's on v3.6.x now :D – gustavohenke Jun 09 '13 at 01:14
  • @gustavohenke yeah although when I update mongoose I get an error that I don't feel like debugging now haha – Justin Meltzer Jun 09 '13 at 01:16
  • you should be running any sharded cluster with three config servers - it appears you only have one... – Asya Kamsky Jun 10 '13 at 04:26
  • also why aren't you just giving your application multiple mongos addresses (for each of your mongos) - it will then failover to the next available mongos if the one it's talking to goes away - you need to be on a relatively recent driver version for that feature. – Asya Kamsky Jun 10 '13 at 04:28
  • @AsyaKamsky I'm using a domain name that resolves to both of my mongos' ip addresses. This won't work? – Justin Meltzer Jun 10 '13 at 14:39
  • @AsyaKamsky Also I'm only on one shard, so does the number of config servers matter in this case? – Justin Meltzer Jun 10 '13 at 14:40
  • of course it matters - if you lose the one config server you won't be able to access your data since mongos won't know even that you have one shard - mongos doesn't store its own data, it uses information that's stored in configdb – Asya Kamsky Jun 10 '13 at 14:58
  • @AsyaKamsky ok thanks. And what about the domain name that resolves to both mongos' ip addresses? is that sufficient? – Justin Meltzer Jun 10 '13 at 15:08
  • I guess I don't see how that can work. if you connect to that hostname which ip address do you actually end up on? – Asya Kamsky Jun 10 '13 at 17:23
  • @AsyaKamsky It does round robin. – Justin Meltzer Jun 10 '13 at 17:33
  • @AsyaKamsky And when one server goes down, I automatically remove the failed IP address from the DNS server's A record. – Justin Meltzer Jun 10 '13 at 17:34
  • but your client does not re-query DNS when that happens. that seems to be a complex and manual way of doing failover - why not just give both server's host:port (for mongos) to your application? – Asya Kamsky Jun 10 '13 at 18:32
  • @AsyaKamsky Why doesn't the client re-query DNS? Is it being cached somewhere? It's just a lot easier to give a domain name because IP addresses can change. – Justin Meltzer Jun 10 '13 at 18:36
  • you *should* use dns names and not ip addresses - I'm just saying you should give individual names to each mongos and give a list of those to your app server to connect to. – Asya Kamsky Jun 10 '13 at 19:58
  • @AsyaKamsky Ok. But that still doesn't explain why the mongodb node.js client isn't receiving an "error" or "close" event when the instance is shut down. Doesn't the client need to get one of those events so it knows to reconnect? – Justin Meltzer Jun 10 '13 at 20:33
  • ensure you have the latest driver as there are several bug fixes related to this. – christkv Aug 12 '13 at 14:06
  • @christkv Thanks. Would you be able to point to some of the relevant bugs in Github issues? – Justin Meltzer Aug 12 '13 at 21:32

0 Answers0