0

I have trouble starting my Hadoop data-node. I did all the research that I could and none of the methods were helpful in solving my issue. Here's my terminal console output when I try to start it using 

hadoop datanode -start

This is what happens:

root@Itanium:~/Desktop/hadoop# hadoop datanode -start
Warning: $HADOOP_HOME is deprecated.

13/09/29 22:11:42 INFO datanode.DataNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = Itanium/127.0.1.1
STARTUP_MSG:   args = [-start]
STARTUP_MSG:   version = 1.2.1
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG:   java = 1.7.0_25
************************************************************/
Usage: java DataNode
           [-rollback]
13/09/29 22:11:42 INFO datanode.DataNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at Itanium/127.0.1.1
************************************************************/
root@Itanium:~/Desktop/hadoop# jps
31438 SecondaryNameNode
32013 Jps
31818 TaskTracker
1146 Bootstrap
    31565 JobTracker
    30930 NameNode

root@Itanium:~/Desktop/hadoop#

As we can see the DataNode attempts to start but then shuts down. All the while I have been having trouble with NameNode starting up. I used to fix this by manually starting it using 

start-dfs.sh

And now the problem is with DataNode. I really would appreciate all your help in resolving this issue.

And one more generic question. Why is Hadoop displaying such inconsistent behavior. I am sure I did not change any of the *-site.xml settings.

Ace
  • 1,501
  • 4
  • 30
  • 49
  • Could you please show me your DN log file? – Tariq Sep 30 '13 at 10:25
  • 1
    Ok, I did my research and looked into the logs and it seems to be an issue with namespaceIDs not matching. Looks like it is a well documented issue, but just for the records I solved it by deleting the ....hadoop/tmp/dfs/name/...." directory...... That solved the issue – Ace Oct 01 '13 at 15:02
  • @Anum Saumithri - Please post your solution as the answer to this question and indicate the exception you are seeing in the logs in your question. I just had this issue and was able to solve it using the hint your comment provided. – Krish Srinivasan Nov 05 '14 at 16:23

3 Answers3

1

use this command hadoop datanode -rollback

Alkesh_IT
  • 376
  • 1
  • 9
1

I had a similar issue as well. Looking at the comment posted by Anup "seems to be an issue with namespaceIDs not matching" I was able to find a reference that showed me how to solve my issue. http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/#caveats

I took a look at the logfile on the slave nodes where the DataNodes did not start. They both had the following exception :

2014-11-05 10:26:14,289 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /scratch/hdfs/data/srinivasand: namenode namespaceID = 1296690356; datanode namespaceID = 1228298945                                                                                                                                   
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)                                                                                    
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)                                                                           
        at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)                                                                                         
        at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)                                                                                                
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)                                                                                         
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)                                                                                  
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)

Fixing this exception solved the issue.

The fix is to either a) delete the dfs data directory. reformat using namenode -format. b) update the VERSION file so that the two namespace IDs match.

I was able to use option b) and the datanodes started successfully after that.

The bug report that leads to this issue is recorded at : https://issues.apache.org/jira/browse/HDFS-107

Krish Srinivasan
  • 568
  • 1
  • 6
  • 14
0

I ever got the same issue, it turns out that 50010 port is occupied by other application, stop the application, restart Hadoop

silverbull
  • 49
  • 3