I don't know why this is occurring, but occasionally I will get this series of repeating errors and the zookeeper instances will go into a bad state.
Tue Feb 16 07:05:04 EST 2016 ERROR ZooKeeper Server: Using config: /opt/zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
Tue Feb 16 07:05:04 EST 2016 ERROR ZooKeeper Server: JMX enabled by default
Tue Feb 16 07:05:04 EST 2016 INFO Process started via: /opt/zookeeper/zookeeper-3.4.6/bin/zkServer.sh
Tue Feb 16 07:05:03 EST 2016 INFO Kill attempted result: 0
Tue Feb 16 07:05:03 EST 2016 INFO Attempting to start/restart ZooKeeper
Tue Feb 16 07:05:03 EST 2016 INFO Attempting to stop instance
Tue Feb 16 07:05:03 EST 2016 INFO Restarting down/not-serving ZooKeeper after 60037 ms pause
Tue Feb 16 07:04:33 EST 2016 INFO ZooKeeper down/not-serving waiting 30026 of 40000 ms before restarting
Tue Feb 16 07:04:05 EST 2016 INFO ZooKeeper Server: Starting zookeeper ... STARTED
Tue Feb 16 07:04:04 EST 2016 ERROR ZooKeeper Server: Using config: /opt/zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
Tue Feb 16 07:04:04 EST 2016 ERROR ZooKeeper Server: JMX enabled by default
The exhibitor stuff uses shared storage on a NAS. The servers are centOs 6.6. It is a three node ensemble, and the one noticible problem I have seen is that the "ensemble" connection string inside of Exhibitor GUI all of a sudden becomes different between the three nodes (one node may "forget" about some of the other nodes in the ensemble).
I don't even know where to look to dig into these causes. Any help or direction will be greatly appreciated. Its trully odd...
update versions
- zk: 3.4.6
- Exhibitor: 1.5.5
