3

I am currently trying to work on HDFS and HBASE. The Hadoop and HBASE are properly installed on a machine and my application runs perfectly when hosted on the same machine.

But when hosting on another machine. On first hit to HBASE I get an error saying:

org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [sitepulsewebsite] in context with path [/SitePulseWeb] threw exception [Request processing failed; nested exception is javax.jdo.JDODataStoreException
NestedThrowables:org.apache.hadoop.hbase.MasterNotRunningException: localhost:60000] with root cause
org.apache.hadoop.hbase.MasterNotRunningException: localhost:60000

And on the second hit I am getting the exception:

org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [sitepulsewebsite] in context with path [/SitePulseWeb] threw exception [Request processing failed; nested exception is javax.jdo.JDODataStoreException: Failed setting up proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to localhost/127.0.0.1:60020 after attempts=1
NestedThrowables: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to localhost/127.0.0.1:60020 after attempts=1] with root cause
java.net.ConnectException: Connection refused

My hbase-site.xml reads as follow :

<configuration>
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://master:54310/hbase </value>
    <description>
        The directory shared by region servers. Should be
        fully-qualified to
        include the filesystem to use. E.g:
        hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR

    </description>

</property>

<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
    <description>The mode the cluster will be in. Possible values are
        false: standalone and pseudo-distributed setups with managed
        Zookeeper
        true: fully-distributed with unmanaged Zookeeper Quorum (see
        hbase-env.sh)
    </description>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>master</value>
    <description>Comma separated list of servers in the ZooKeeper Quorum.
        If HBASE_MANAGES_ZK is set in hbase-env.sh this is the list of
        servers which we will start/stop ZooKeeper on.
    </description>
</property>
<property>
    <name>hbase.master</name>
    <value>master:60010</value>
</property>
<property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
</property></configuration>

UPDATED LOGS

Looking into the logs (DEBUG Level) created by my Java application, I found the following logs:

1 2012-01-12 17:12:13,328 DEBUG Thread-1320 org.apache.hadoop.ipc.HBaseClient - IPC Client (47) connection to localhost/127.0.0.1:60020 from an unknown user: closed
2 2012-01-12 17:12:13,328 INFO Thread-1320 org.apache.hadoop.ipc.HbaseRPC - Server at localhost/127.0.0.1:60020 could not be reached after 1 tries, giving up.
3 2012-01-12 17:12:13,328 DEBUG Thread-1320 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - locateRegionInMeta parentTable=-ROOT-, metaLocation=address: localhost:60020, regioninfo: -ROOT-,,0.70236052, attempt=0 of 10 failed; retrying after sleep of 1000 because: Failed setting up proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to localhost/127.0.0.1:60020 after attempts=1
4 2012-01-12 17:12:13,328 DEBUG Thread-1320 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@9d1e83; hsa=localhost:60020
5 2012-01-12 17:12:13,736 DEBUG Thread-1268 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@9d1e83; hsa=localhost:60020
6 2012-01-12 17:12:13,736 DEBUG Thread-1268 org.apache.hadoop.ipc.HBaseClient - Connecting to localhost/127.0.0.1:60020
7 2012-01-12 17:12:13,737 DEBUG Thread-1268 org.apache.hadoop.ipc.HBaseClient - closing ipc connection to localhost/127.0.0.1:60020: Connection refused
8 java.net.ConnectException: Connection refused

In /etc/hosts file when the mapping was changed from 

127.0.0.1 localhost

to

<my_server_IP> localhost

My application worked perfectly fine. Hence I need some way to tell the application to connect to desired hostname and not localhost.

I have tried debugging it, without any success.

halfer
  • 19,824
  • 17
  • 99
  • 186
devsri
  • 6,173
  • 5
  • 32
  • 40
  • So I am confused. Did u resolve the problem? It turns out there are a few things that cause that error. – Amir Raminfar Jan 15 '12 at 16:52
  • No I did not resolve, I can not map localhost to the IP of my server as when the application goes on production, other application running on the system expecting localhost to be self system will break. – devsri Jan 16 '12 at 06:53
  • Yes I have disabled on both the systems and also verified by cat /proc/sys/net/ipv6/conf/all/disable_ipv6 , which gives 1 as output – devsri Jan 17 '12 at 09:32

4 Answers4

6

I don't know if this is your problem, but it generally is a problem to use localhost if you are not accessing everything from the same host.

So don't use localhost!

And in general don't change the definition of localhost. Localhost is 127.0.0.1 by defintion.

You define hbase.rootdir as hdfs://master:54310/hbase and hbase.zookeeper.quorum as master.

What is master? It really should be a fully qualified domain name of the main ethernet interface of your host. The reverse DNS of the IP address of that interface should resolve to the same FQDN that you fill in to these fields. (Or just us the raw IP address if you can't control the reverse dns)

Make sure your HDFS configs also use the same FQDN's or IP addresses or synchronized /etc/hosts files. Synchronized /etc/hosts files should make sure the forward and reverse DNS is the same as long as all the hosts (all the HDFS and HBase and your clients) use the same /etc/hosts and there is no OS stuff overriding the /etc/hosts. In general I don't like to do anything with /etc/hosts. It will eventually bite you.

Your remote client should then access your HBase master via the same FQDN or IP address.

I have found that this kind of DNS issues can cause quite a bit of confusion.

If you need a reality check, just use IP addresses everywhere till you make it work. Then experiment with Fully Qualified Domain Names or synchronized /etc/hosts files.

Robert J Berger
  • 890
  • 9
  • 12
  • Thanks a ton........ this was a real big problem......... Now how do I handle this problem of using ip instead of hostname. Also, I left the hbase configs as it is but changed the hbase-site.xml of my application. What is the solution to run it normally with domain name....... Thnks – devsri Jan 19 '12 at 10:11
  • 1
    Its funny, we're revisiting this as we try to get hbase to run as pseudo-distributed mode on a Vagrant instance... The fact that hadoop/hbase figures things out by doing reverse dns lookups on its interfaces makes things confusing when you have multiple interfaces or unusual virtualization environments.. – Robert J Berger Jan 19 '12 at 22:20
  • 1
    But you should be able to set your /etc/hosts everywhere to name the ip address that works. That should drive the forward and reverse DNS properly assuming the ip is associated with the interface being used. I should have some experiments with Vagrant done soon. Will report back – Robert J Berger Jan 19 '12 at 22:22
4

I had something like this in /etc/hosts:

127.0.0.1   localhost
127.0.1.1   <hostname>

I changed it to solve <hostname> to 127.0.0.1 and that seemed to solve the problem.

127.0.0.1   localhost <hostname>
Diego Pino
  • 11,278
  • 1
  • 55
  • 57
1

Many thanks to @Robert J Berger to leading me to the answer to the same issue. I wasn't having issue #1, but I was having issue #2, with the server reporting:

Failed setting up proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to localhost/127.0.0.1:60020 after attempts=1

However, his answer was focusing on the 'master' side so I'd thought I'd expand on his with my own answer. My DNS (used by all machines in the cluster) resolves 'master' so client connection to the server was not the issue (was working regardless of 'master' being only hostname, not FQDN). [I agree w/ Robert that avoiding /etc/hosts modifications is ultimately the most maintainable solution.]

My issue was the server connecting to the client. Hbase is using the hostname of the machine to resolve the IP to bind internal communications to, and it turns out I had this set on master properly but not the nodes.

It was due to how the /etc/hosts file was being built on the regionserver nodes as they were getting issued. The hbase server errors disappeared when i changed /etc/hosts on each node from:

127.0.0.1    node-hostname

to:

<actual ip>  node-hostname

Now the Hbase server sees the nodes properly and can build proxies to each.

mujimu
  • 1,449
  • 11
  • 11
0

Execute following command

sudo hostname thehostnameyouwanttogiveformachine

edit /etc/hosts file

put

actual-ip thehostnameyouwanttogiveformachine

restart the machine

and check

Infinity
  • 683
  • 1
  • 6
  • 17