0

I am trying to start hadoop. But, Datanodes could not be started, and this is the output of the start hdfs command. Although namenode is started correctly.

 Starting namenodes on 
 Starting datanodes
 worker2: bash: line 1: /home/hadoop-master/my-hadoop-install/hadoop-3.3.4/bin/hdfs: No such file or directory
 worker1: bash: line 1: /home/hadoop-master/my-hadoop-install/hadoop-3.3.4/bin/hdfs: No such file or directory
 Starting secondary namenodes [hadoop-master]
Radwa Ahmed
  • 131
  • 1
  • 1
  • 9

1 Answers1

0

It seems hadoop binaries are not present on worker1 and worker2 node and if present then check path once. Go to /home/hadoop-master/my-hadoop-install/hadoop-3.3.4/bin path in worker1 and worker2 node.

Uddhav Savani
  • 31
  • 6
  • The above problem was because data node machines' usernames were different from each others. Unfortunately, after i have solved the problem, I have encountered a new problem that website does not show up datanodes "like they had never been started", Although when i run jps command, it shows datanode. Also, Data folder does not have any files or directories for the datanode. – Radwa Ahmed Mar 20 '23 at 22:19
  • Here is the one of many scenario which can lead to that, When we start datanode daemons, after all initialization it will continuesly try to contact namenode provided in configurations, but if it is not able to contact the namenode then datanode daemon is running so you can see it in jps, but it is not registered to namenode. And the ui is served by namenode so you will not be able to see datanode in it. You can verify it from datanode logs if this situation is occuring or not. Firewalls may cause this problems so you can also check 9000 port is open or not in namenode node. – Uddhav Savani Mar 21 '23 at 04:02
  • I have allowed all required ports. But, datanode logs show errors connecting to the server. – Radwa Ahmed Mar 21 '23 at 11:08
  • Can you modify or post new question for it including core-site.xml, hdfs-site.xml, "/etc/hosts" file, and error message in datanode logs, provide details of the datanode which is not connecting – Uddhav Savani Mar 21 '23 at 11:33
  • I have already posted a new question. this is the link of it: https://stackoverflow.com/questions/75796112/datanode-directory-is-empty – Radwa Ahmed Mar 21 '23 at 11:54
  • Are You using Cloud VM?? If yes then make sure you have inbound and outbound port rules set for port 9000 and i guess only opening it from cloud portal will not work you have to enable it from firewall in os. If your OS is linux base then can you give what is the output of "nc -v 9000", it requires netcat package – Uddhav Savani Mar 21 '23 at 12:10
  • yes, i am using cloud VM. Output is "nc: connect to 64.226.67.23 port 9000 (tcp) failed: Connection refused" – Radwa Ahmed Mar 21 '23 at 12:19
  • Also, I have allowed the port from the machine terminal, and this is part of the "ufw status verbose command" output: "9000 (v6) ALLOW IN Anywhere (v6)" – Radwa Ahmed Mar 21 '23 at 12:28
  • Was namenode daemon running when you tried nc command? if yes then you have firewall and port configuration problem only nc command should give you output connected. If possible try with private IPs once if your nodes are in one vpc. Also if possible provide "netstat -anp | grep -i 9000" commands output in namenode – Uddhav Savani Mar 21 '23 at 12:46