Hbase master not able to start regionserver

Question

I have one master and one regionserer running on one machine,now I want to add another region server to it.

This new machine has all the connection config required (as in passwordless ssh etc), as these two machines are running datanodes for my underlying Hadoop (and start/stop from hadoop master starts/stops datanodes on all the machines).

I have mentioned two servers (with their entries in /etc/hosts):

master
slave

Now if I do ./start-hbase.sh on the master In the console it says "starting regionserver on slave", but the corresponding log is empty and in the web console i do not see the slave as one of the regionservers (master region server starts fine).

So I thought maybe I will start the region server manually on the slave and I did

./hbase-daemon.sh start regionserver

The region server starts but in the logs I see:

2011-09-05 14:28:54,331 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at master:60000
2011-09-05 14:28:54,501 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Connected to master at master:60000
2011-09-05 14:28:54,672 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at master:60000 that we are up
2011-09-05 14:28:54,757 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: error telling master we are up
java.io.IOException: Call to master/174.129.183.20:60000 failed on local exception: java.io.EOFException
        at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:806)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
        at $Proxy5.regionServerStartup(Unknown Source)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1538)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:375)
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:539)
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:477)

I thought may be if I restarted the hbase master it will recognize the regionserver, but when I do that I get an error "Region server started on slave with pid:xxx, stop it first".

What am I missing here, I am using exactly the same hbase-site.xml file on both the machines. Please help

Try this. Stop everything first. From the master, do ./stop-hbase.sh. Now start everything, using ./start-hbase.sh from the master. On the master box, do $JAVA_HOME/bin/jps (replace JAVA_HOME with java you're using for your HBase installation). This will list all java processes running. You should see your Hadoop datanode, namenode listed, and then HMaster. Do the same from regionserver box and you should see HRegionserver listed. Try this and let us know if you're still seeing errors in the logs and what those errors are. — knt, Sep 06 '11 at 12:35

score 0 · Answer 1 · answered Apr 25 '12 at 12:49

If any where you have localhost first this remove that from hadoop and hbase conf files.

add your ipaddress somenameforthemachine

then use somenameforthemachine for all configuration in hadoop, so with this configuration make hadoop and hbase running on single machine, so after everything is working fine like you can see the processes

HQuorumPeer 6261 HRegionServer 4100 TaskTracker 6000 HMaster 7701 Main 7765 Main 3426 DataNode 11851 Main 3149 NameNode 3712 SecondaryNameNode 12548 Jps 6929 Main 2688 Main 3820 JobTracker

then add your slave machine to slave with domainname (You can specify in host name as ipaddress master ipaddress slave )

ping master from both master slave if you are able to ping its fine then also ping ipaddress from both machine so if this process runs fine proceed.

copy the same hadoop and hbase folder to the slave in same structure you are having in master suppose you have it in /home/username so it should be in same place on slave.

try this will solve your problem.. this will generally happens because of host configuration the ip and port configured increctly, tell if it solves your problem.

http://stackoverflow.com/questions/9615707/hbase-how-to-specify-hostname-for-hbase-master/10447844#10447844 — Infinity, May 04 '12 at 11:33

score 0 · Answer 2 · answered Dec 08 '11 at 23:55

0

Try to verify your DNS settings.Hbase requires both forward and reverse resolving

answered Dec 08 '11 at 23:55

Hbase master not able to start regionserver

2 Answers2