We have a Mongodb replicaset with 2 nodes, each of them located in a distinct CPD, in different countries.
The latency between both nodes is measurable. This is not a problem when writing data, as this is done once a day using an offline process.
But when reading, in each request the nodes stablish connections between them, which take quite a lot of time and make the request slow, and create problems on the network.
We've found ~700 connections open between the primary and the secondary nodes.
I've tried several connection, connection string and readPreference settings but this always happens.
Is there any way that no connections are done between nodes on each request? How do people cope with this problem when the nodes are not so close between them?
EDIT:
If I run the test script repeately, sometimes it's quite fast. I see this in the MongoLog details:
180 [+0.004990] IO FINE getting cursor body
181 [+0.005032] CON FINE No timeout changes for xxxx:27017;rs1;.;101877
182 [+0.005080] CON FINE Initializing cursor timeout to 30000 (from connection options)
183 [+0.005095] CON INFO command supports Read Preferences
184 [+0.005109] CON INFO mongo_get_read_write_connection: finding a REPLSET connection (read)
185 [+0.005128] CON FINE found connection xxxxx:27017;rs1;.;101877 (looking for xxxx:27017;rs1;.;101877)
186 [+0.005140] CON FINE is_ping: skipping: last ran at 1432113918, now: 1432113922, time left: 1
187 [+0.005153] CON FINE found connection yyyy:27017;rs1;.;101877 (looking for yyyyy:27017;rs1;.;101877)
188 [+0.005163] CON FINE is_ping: skipping: last ran at 1432113919, now: 1432113922, time left: 2
189 [+0.005175] CON FINE discover_topology: checking ismaster for xxxxx:27017;rs1;.;101877
190 [+0.005186] CON FINE found connection xxxx:27017;rs1;.;101877 (looking for xxxxx:27017;rs1;.;101877)
191 [+0.005196] CON FINE ismaster: skipping: last ran at 1432113919, now: 1432113922, time left: 12
192 [+0.005206] CON FINE discover_topology: ismaster got skipped
193 [+0.005218] CON FINE discover_topology: checking ismaster for yyyyy:27017;rs1;.;101877
194 [+0.005228] CON FINE found connection yyyyy:27017;rs1;.;101877 (looking for yyyyy:27017;rs1;.;101877)
195 [+0.005237] CON FINE ismaster: skipping: last ran at 1432113919, now: 1432113922, time left: 12
196 [+0.005246] CON FINE discover_topology: ismaster got skipped
When any of the 3 timers reach 0 then connections are stablished. Looks like I could increment the intervals for the 3 checks, right?