We are deploying mongodb cluster through ansible on AWS EC2 instance. Once a DB instance is provisioned, we give it corresponding tag for Primary / Secondary. We have 3 nodes - 1 for Primary and 2 for secondary.
While running the ansible scripts for the first time, we are choosing the primary instance by Tag type, initiating it and adding secondary nodes to it
rs.initiate()
rs.add(secodnaryHost1)
rs.add(secondaryHost2)
This works fine.
However at times, when one of the secondary becomes the primary, our instance Tag type - primary and mongoDB - primary are not the same. At such a scenario, if we run our ansible scripts, they fail because the primary tag type (which is now a secondary) can't run above commands.
What are the best ways to handles such mismatch?