I'm trying to build a brain split resolver for akka cluster. But it's quite hard to simulate the scenario where certain nodes are unreachable from the rest of the origin cluster and form their own cluster.
It can't be done by stoping and restart nodes because the newly started nodes are not the same one since UID of those nodes have changed.
network failure injection in test conductor seems quite promising but I didn't find any doc about it. So, How can I create a network partition scenario?
I was recently writing split brain resolver and these are the tips for testing, which I found useful:
Before making full simulation it's maybe worth to cover some parts of the code with unit tests. Akka project itself makes something similar for AutoDown feature. You just emulate events and send messages to DowningProvider and check, that it replies accordingly. It helps to test some corner cases without a complicated setup.
To simulate cluster and network splits I used multi-jvm testing. Check out Akka's SplitBrainSpec or WeeklyUpSpec. They are sometimes much more helpful than documentation. For creating network split testConductor.blackhole should be used and for removing the split testConductor.passThrough. I found it important to always put barriers and constantly check the current cluster state (before running the actual test wait for all members to be up/join cluster/see network split).
