1

In my setup, Because of Power interruption My VMs got down state after restarting all my VMs I cant able to start Hbase, all other services can restart/stop/start but in UI for CM in hbase ACTIONS tab all options were hidden..

Following is my hbase master log,now its in exiting mode

6:29:05.880 AM  ERROR   org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
ZooKeeper exists failed after 3 retries
6:29:05.880 AM  WARN    org.apache.hadoop.hbase.zookeeper.ZKUtil    
hconnection Unable to set watcher on znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:420)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:621)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:684)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:126)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.stopMaster(HMasterCommandLine.java:175)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:106)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1962)
6:29:05.880 AM  ERROR   org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher  
hconnection Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:420)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:621)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:684)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:126)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.stopMaster(HMasterCommandLine.java:175)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:106)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1962)
6:29:05.881 AM  INFO    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation 
This client just lost it's session with ZooKeeper, will automatically reconnect when needed.
6:29:06.880 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderaalpha.alp.com/10.0.0.13:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:06.882 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:06.983 AM  WARN    org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
6:29:06.983 AM  INFO    org.apache.hadoop.hbase.util.RetryCounter   
Sleeping 2000ms before retry #1...
6:29:06.983 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn1.alp.com/10.0.0.14:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:06.984 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:07.084 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn3.alp.com/10.0.0.16:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:27.103 AM  INFO    org.apache.zookeeper.ClientCnxn 
Client session timed out, have not heard from server in 20019ms for sessionid 0x0, closing socket connection and attempting reconnect
6:29:27.204 AM  WARN    org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
6:29:27.204 AM  INFO    org.apache.hadoop.hbase.util.RetryCounter   
Sleeping 4000ms before retry #2...
6:29:28.204 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderaalpha.alp.com/10.0.0.13:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:28.206 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:28.306 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn1.alp.com/10.0.0.14:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:28.307 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:28.408 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn3.alp.com/10.0.0.16:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:48.425 AM  INFO    org.apache.zookeeper.ClientCnxn 
Client session timed out, have not heard from server in 20017ms for sessionid 0x0, closing socket connection and attempting reconnect
6:29:48.526 AM  WARN    org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
6:29:48.526 AM  INFO    org.apache.hadoop.hbase.util.RetryCounter   
Sleeping 8000ms before retry #3...
6:29:49.526 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderaalpha.alp.com/10.0.0.13:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:49.529 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:49.630 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn1.alp.com/10.0.0.14:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:49.630 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:49.731 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn3.alp.com/10.0.0.16:2181. Will not attempt to authenticate using SASL (unknown error)
6:30:09.744 AM  INFO    org.apache.zookeeper.ClientCnxn 
Client session timed out, have not heard from server in 20013ms for sessionid 0x0, closing socket connection and attempting reconnect
6:30:09.845 AM  WARN    org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
6:30:09.845 AM  ERROR   org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
ZooKeeper exists failed after 3 retries

Thanks for your attention...

Narenthirasamy Ramasamy
  • 65
  • 1
  • 14

2 Answers2

0

From the log we can see it failed to connect ZooKeeper server. You may need to start ZooKeeper service and try to restart HBase

zhutoulala
  • 4,792
  • 2
  • 21
  • 34
0

As zhutoulala said before, I would recommend that too, then, if it's still not working, you should probably delete the znode of HBase:

Use the 'hbase zkcli' command from one of the servers running an HBase service. Once in the ZooKeeper CLI, you can run 'rmr /hbase' to remove the znode.

kulssaka
  • 226
  • 8
  • 27