17

I am trying to setup the multinode cluster of Hbase. When i do the jps on slave i get

5780 Jps
5558 HQuorumPeer
5684 HRegionServer
1963 DataNode
2093 TaskTracker

similarly on master i get 

4254 SecondaryNameNode
15226 Jps
14982 HMaster
3907 NameNode
14921 HQuorumPeer
4340 JobTracker

EVerything is runnnig properly. But when i try to create table on hbase shell. It gives an error 

ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

regionserver log of my slave(where region server is running):

2013-06-11 13:09:53,119 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at localhost,60000,137093$
2013-06-11 13:10:53,190 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: localhost/127.0.0.1:60000
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:425)
        at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
        at $Proxy8.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2037)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2083)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:744)
        at java.lang.Thread.run(Thread.java:722)
2013-06-11 13:10:53,391 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at localhost,60000,137093$

FYI, i have also took care of /etc/hosts file on both master and slave.

127.0.0.1       localhost
127.0.0.1       naresh-PC

I again did changes in /etc/hosts file 127.0.1.1 to naresh-PC. But still getting this error

2013-06-11 14:51:17,781 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at naresh-pc,60000,137094$
2013-06-11 14:52:17,817 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.UnknownHostException: unknown host: naresh-pc
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.<init>(HBaseClient.java:276)
        at org.apache.hadoop.hbase.ipc.HBaseClient.createConnection(HBaseClient.java:255)
        at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1111)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
        at $Proxy8.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2037)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2083)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:744)
        at java.lang.Thread.run(Thread.java:722)
Prasad Khode
  • 6,602
  • 11
  • 44
  • 59
Naresh
  • 5,073
  • 12
  • 67
  • 124

7 Answers7

22

Try clearing all the states in Zookeeper.

  • Stop Zookeeper
  • Wipe the Zookeeper data directory
  • Start Zookeeper

I was getting the same issue and followed this approach and it worked fine.

Naman
  • 27,789
  • 26
  • 218
  • 353
Nishu Tayal
  • 20,106
  • 8
  • 49
  • 101
  • 8
    Can you please explain how to perform these steps on Ubuntu. Can you list the commands that we need to follow. – User7723337 Apr 10 '14 at 09:24
  • 1
    Ran into this today and wiping the zk data directory also worked for me. Nishu, your hbase-site.xml file should have a hbase.zookeeper.property.dataDir property that tells where this path is. Just make sure hbase is down before clearing it. – dnuttle Feb 28 '15 at 21:39
  • It is enough just to delete `/hbase` znode in ZooKeeper via zkCli.sh with command `rmr /hbase`. No need to delete whole ZooKeeper directory. Be sure HBase Masters are not running during znode deletion. – 32cupo May 17 '19 at 15:31
4

You need to change the configuration on the slave node to point at the master. It is currently pointing to localhost and not connecting to the actual master:

"org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: localhost/127.0.0.1:60000 at "

Arnon Rotem-Gal-Oz
  • 25,469
  • 3
  • 45
  • 68
  • well, i did give master's ip but still get the same problem. org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: master:60000 at – Naresh Jun 19 '13 at 08:50
  • After setting ` hbase.master ip-hmaster:60000 ` in `/usr/lib/hbase/conf/hbase-site.xml` on all slave nodes the issue was solved. Thank you for the answer. – 030 Jun 04 '14 at 13:25
1

I'm hosting my own cluster inside Docker. Here's what worked in my case. I grepped the HBase log file for errors and found "Master passed us a different hostname to use" 

`[root@docker-iop bin]# grep ERROR /var/log/hbase/hbase-hbase-regionserver-bi-mgmt01.local.log
2016-10-06 00:05:29,816 ERROR [regionserver/bi-mgmt01.local/111.11.2.3:16020] regionserver.HRegionServer: Master passed us a different hostname to use; was=my-host-name, but now=111.22.33.444'

I mapped my-host-name to 111.22.333.444 in my hosts file, restarted HBase and it worked.

s d
  • 2,666
  • 4
  • 26
  • 42
  • See also https://issues.apache.org/jira/browse/HBASE-3431, https://www.syncfusion.com/kb/7099/how-to-avoid-hbase-services-to-bind-on-multi-hosts, https://issues.apache.org/jira/browse/HBASE-12954, https://issues.apache.org/jira/browse/HBASE-18226 and http://www-01.ibm.com/support/docview.wss?uid=swg21986077 – Joe C Nov 27 '18 at 06:58
0

I also had the same issue with a fully distributed hbase cluster with the configuration below.

  1. Master Node (Node-A)
  2. Backup Masters ($HBASE_HOME/conf/backup-masters) (Node-B & Node-C)
  3. 3 Replication servers (Node-A, Node-B & Node-C)

RCA: The backup-masters nodes were attempted to be started when the cluster started.

Solution I removed the backup masters by making $HBASE_HOME/conf/backup-masters empty in all hbase nodes.

So I had a cluster running without backup masters.

I wonder if the master node and master nodes must not also function as regionservers? The HBase documentation says otherwise though.

Abdurrahman Adebiyi
  • 470
  • 2
  • 12
0

I came across the same issue and could not find anything, it turns out I was copy pasting from the Hbase documentation (https://hbase.apache.org/book.html#shell_exercises). I believe some character in there may be creating the error, so try to manually enter:

create 'test', 'cf'

Olivier Maxwell
  • 65
  • 1
  • 9
0

We resolved this issue. Solution is to

  • stop Hbase
  • log to zookeeper-client as root
  • execute command rmr /hbase-unsecure/meta-region-server
  • start Hbase

We stop/start Hbase through Ambari UI, delete /hbase... through server bash shell.

[root@s1 ~]# zookeeper-client
Connecting to localhost:2181
.......
[zk: localhost:2181(CONNECTED) 0] rmr /hbase-unsecure/meta-region-server
Songlet Li
  • 1
  • 2
0

I use docker/docker-compose to set up my distributed hbase, after I make changes, I can not create table in hbase shell.

I docker rm all the related images, and rebuild them. It works. Also, simply rebuilding the images doesn't work...

sch001
  • 71
  • 4