10

I'm trying to set up a Cloudera Hadoop cluster, with a master node containing the namenode, secondarynamenode and jobtracker, and two others nodes containing the datanode and tasktracker. The Cloudera version is 4.6, the OS is ubuntu precise x64. Also, this cluster is being created from a AWS instance. ssh passwordless has been set as well, Java instalation Oracle-7.

Whenever I execute sudo service hadoop-hdfs-namenode start I get:

2014-05-14 05:08:38,023 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
        at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:329)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:317)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.getRpcServerAddress(NameNode.java:370)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:422)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:442)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:621)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:606)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1177)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1241)

My core-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->
<configuration>
   <property>
      <name>fs.defaultFS</name>
      <value>hdfs://<master-ip>:8020</value>
   </property>
</configuration>

mapred-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->
<configuration>
   <property>
      <name>mapred.job.tracker</name>
      <value>hdfs://<master-ip>:8021</value>
   </property>
</configuration>

hdfs-site.xml:

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->
<configuration>
   <property>
      <name>dfs.replication</name>
      <value>2</value>
   </property>
   <property>
      <name>dfs.permissions</name>
      <value>false</value>
   </property>
</configuration>

I tried using public ip, private-ip, public dns and fqdn, but the result is the same. The directory /etc/hadoop/conf.empty looks like:

-rw-r--r-- 1 root root   2998 Feb 26 10:21 capacity-scheduler.xml
-rw-r--r-- 1 root hadoop 1335 Feb 26 10:21 configuration.xsl
-rw-r--r-- 1 root root    233 Feb 26 10:21 container-executor.cfg
-rwxr-xr-x 1 root root    287 May 14 05:09 core-site.xml
-rwxr-xr-x 1 root root   2445 May 14 05:09 hadoop-env.sh
-rw-r--r-- 1 root hadoop 1774 Feb 26 10:21 hadoop-metrics2.properties
-rw-r--r-- 1 root hadoop 2490 Feb 26 10:21 hadoop-metrics.properties
-rw-r--r-- 1 root hadoop 9196 Feb 26 10:21 hadoop-policy.xml
-rwxr-xr-x 1 root root    332 May 14 05:09 hdfs-site.xml
-rw-r--r-- 1 root hadoop 8735 Feb 26 10:21 log4j.properties
-rw-r--r-- 1 root root   4113 Feb 26 10:21 mapred-queues.xml.template
-rwxr-xr-x 1 root root    290 May 14 05:09 mapred-site.xml
-rw-r--r-- 1 root root    178 Feb 26 10:21 mapred-site.xml.template
-rwxr-xr-x 1 root root     12 May 14 05:09 masters
-rwxr-xr-x 1 root root     29 May 14 05:09 slaves
-rw-r--r-- 1 root hadoop 2316 Feb 26 10:21 ssl-client.xml.example
-rw-r--r-- 1 root hadoop 2251 Feb 26 10:21 ssl-server.xml.example
-rw-r--r-- 1 root root   2513 Feb 26 10:21 yarn-env.sh
-rw-r--r-- 1 root root   2262 Feb 26 10:21 yarn-site.xml

and slaves lists the ip addresses of the two slave machines:

<slave1-ip>
<slave2-ip>

Executing 

update-alternatives --get-selections | grep hadoop
hadoop-conf                    auto     /etc/hadoop/conf.empty

I've done a lot of search, but didn't get anything that could help me fix my problem. Could someone offer any clue what's going on?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
cybertextron
  • 10,547
  • 28
  • 104
  • 208
  • Ok..I haven't used cloudera much..but,why fs.defaultFS and not fs.default.name? I believe fs.defaultFS is used for HDFS HA. – Chandra kant May 14 '14 at 16:02
  • @Chandrakant It fails for both `fs.defaultFS` and `fs.default.name`. Furthermore, in the Cloudera page it says `fs.default.name` is deprecated. – cybertextron May 14 '14 at 16:09
  • I know that fs.default.name is deprecated in favor of fs.defaultFS . I just didn't see that you were using hadoop-2 in the first place. Looked at the namenode formatting line and realized. – Chandra kant May 14 '14 at 16:28
  • @Chandrakant Do you have a clue why is the namenode not starting? – cybertextron May 14 '14 at 17:17
  • Generally "No authority" means no permission. Have you checked the permissions of the directories namenode is writing to? – Chandra kant May 15 '14 at 03:42
  • Also can you double check the configuration settings on datanodes and tasktrackers.. – Chandra kant May 15 '14 at 03:46

4 Answers4

16

I was facing the same issue and fixed by formatting the namenode. Below is the command:

hdfs namenode -format

core-site.xml entry is :

<configuration>
   <property>
      <name>fs.defaultFS</name>
      <value>hdfs://localhost:9000</value>
   </property>
</configuration>

That will definitely solve the problem.

Sonu
  • 712
  • 9
  • 7
  • Had to update my hadoop-3.1.2\etc\hadoop\core-site.xml with this and it worked perfectly in Standalone mode, thanks! Also had to download a DLL from following the links on here and put the DLL in my bin directory: https://wiki.apache.org/hadoop/WindowsProblems – abelito Apr 05 '19 at 07:15
4

I ran into this same thing. I found I had to add a fs.defaultFS property to hdfs-site.xml to match the fs.defaultFS property in core-site.xml:

<property>
      <name>fs.defaultFS</name>
      <value>hdfs://<master-ip>:8020</value>
</property>

Once I added this, the secondary namenode started OK.

Nik Bates-Haus
  • 211
  • 2
  • 4
1

Make sure you have set the HADOOP_PREFIX variable correctly as indicated in the link: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html

Even i faced the same issue as yours and it got rectified by setting this variable

Vishwanath S
  • 37
  • 3
0

Might be you had given a wrong syntax for dfs.datanode.data.dir or dfs.namenode.data.dir in hdfs-site.xml. If you miss / in the value you will get this error. Check the syntax of file:///home/hadoop/hdfs/

pavanpenjandra
  • 1
  • 1