0

I need to do data analysis using Hadoop. Therefore I have installed Hadoop and configured as below. But localhost:9870 is not working. Even I have format namenode every time I worked with that. Some articles and answers of this forum mentioned that 9870 is the updated one from 50070. I have win 10. I also referred answers in this forum but none of them worked. Java-home and hadoop-home paths are set. Paths to bin and sbin of hadoop are also set up. Can anyone please tell me what I am doing wrong in here?

I referred this site to do the installation and configuration. https://medium.com/@pedro.a.hdez.a/hadoop-3-2-2-installation-guide-for-windows-10-454f5b5c22d3

core-site.xml

I have set up the Java path in this xml as well.

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://localhost:9870</value>
</property>

hdfs-site.xml

<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>
<property>
  <name>dfs.namenode.name.dir</name>
  <value>C:\hadoop-3.2.2\data\namenode</value>
</property>
<property>
  <name>dfs.datanode.data.dir</name>
  <value>C:\hadoop-3.2.2\data\datanode</value>
</property>

mapred-site.xml

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

yarn-site.xml

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Virgil
  • 3
  • 1
  • 4

1 Answers1

0

If you look at the namenode logs, it very likely has an error saying something about a port already being in use.

The default fs.defaultFS port should be 9000 - https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html ; you shouldn't change this without good reason.

The Namenode web UI isn't the value in fs.defaultFS. It's default port is 9870, and is defined by dfs.namenode.http-address in hdfs-site.xml

need to do data analysis

You can do analysis on Windows without Hadoop using Spark, Hive, MapReduce, etc. directly and it'll have direct access to your machine without being limited by YARN container sizes.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • I edited the change of 9000. Also can you please tell me where do I put this "dfs.namenode.http-address in hdfs-site.xml". In apache docs it says loclhost:9870 can be browsed to get the web interface. But it can't be reached when I browse the address. Thank you for your time. @OneCricketeer – Virgil Apr 25 '22 at 17:19
  • You don't need to set the http address for the namenode yourself. It's already defined as a default. But you can _override_ it just like you did for the others - add a new `` tag. You need to look at the process logs to know if it really started the server – OneCricketeer Apr 25 '22 at 19:09