I am working with a Cassandra cluster consisting of 9 nodes across 3 data centers. 3 nodes per DC.
We recently found that the system_auth keyspace was still set to SimpleStrategy and we were using the Cassandra default user.
Following the steps to set up NetworkTopologyStrategy with a replication factor of 3, we logged in and issued the alter keyspace command.
ALTER KEYSPACE system_auth WITH replication = { 'class' : 'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 3, 'dc3' : 3 }
This came back with an error in the logs
CustomTThreadPoolServer.java:223 - Error occurred during processing of message.
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
After running this command we were unable to log in to cqlsh and nodetool repair system_auth returned that there was nothing to repair.
nodetool status system_auth showed one data center with each node at 100% ownership and the other 2 data centers showing 0%
Any thoughts on why this is happening? I performed this exact change in 2 of my lower environments that have the same configuration of servers and the issue never came up.
