Scenario:
Have a MongoDB replicaset cluster with 3 nodes. Just before issuing index build command state was as following: NODE 1: SECONDARY NODE 2: SECONDARY NODE 3: PRIMARY
At moment 1, issued command on NODE 3 (PRIMARY) to create index:
db.scores.createIndex({member_id: 1, metric_name: 1, score_id: 1, "writes.k": 1}, {background: true});
Index build was running in background mode for about 2h17min, after which I accidentally pressed CTRL+C in SSH session and killed process. It kept running few more minutes after that in the background (I could witness in Kibana logs), but it then got halted. Because mongod has created locks on NODE 3, had to restart docker container in NODE 3 (PRIMARY) in which mongod is running. Because of the restart, other two nodes in the cluster elected NODE 1 as a new PRIMARY. When NODE 3 got back to normal, I couldn't connect to mongo shell, but Kibana logs were showing that it picked up index build in the foreground and kept running away with it. We let it finish.
Current state:
- Only NODE 3 (still SECONDARY) has index named
'member_id_1_metric_name_1_score_id_1_writes.k_1'. Attempted to delete index on NODE 3 gives error:
SECONDARY> db.scores.dropIndex('member_id_1_metric_name_1_score_id_1_writes.k_1') { "ok" : 0, "errmsg" : "not master", "code" : 10107, "codeName" : "NotMaster" }Attempted to delete index on NODE 1 (PRIMARY) gives error:
db.scores.dropIndex('member_id_1_metric_name_1_score_id_1_writes.k_1') { "nIndexesWas" : 4, "ok" : 0, "errmsg" : "index not found with name [member_id_1_metric_name_1_score_id_1_writes.k_1]", "code" : 27, "codeName" : "IndexNotFound" }
What to do? What are the options to remove this index and attempt again from new PRIMARY (NODE 1)? What are the consequences of having this index sitting like this and only present on one SECONDARY node in replicaset? How bad is this?
Thanks!
