Zookeeper admin guide say the following:
A server might not be able to read its database and fail to come up because of some file corruption in the transaction logs of the ZooKeeper server. You will see some IOException on loading ZooKeeper database.
in such a case, make sure all the other servers in your ensemble are up and working. Use "stat" command on the command port to see if they are in good health. After you have verified that all the other servers of the ensemble are up, you can go ahead and clean the database of the corrupt server. Delete all the files in datadir/version-2 and datalogdir/version-2/. Restart the server.
How do I know from the information below when it's safe for current Zookeeper Server to restart with an empty state ?
For follower
Latency min/avg/max: 0/0/1891
Received: 334210
Sent: 333068
Connections: 23
Outstanding: 0
Zxid: 0x59000712a0
Mode: follower
Node count: 281908
For leader
Latency min/avg/max: 0/0/1829
Received: 9258940
Sent: 9252915
Connections: 829
Outstanding: 2
Zxid: 0x5900074b05
Mode: leader
Node count: 281909
Also would trying to connect to current ensemble and try to create a session using Zookeeper client library also be a good way to verify the ensemble currently have quorum and is safe to join with empty state?
