A DataNode stores data in the HFS (HadoopFileSystem). A functional filesystem has more than one DataNode, with data replicated across them.
Questions tagged [datanode]
86 questions
0
votes
0 answers
NameNode and Datanode not starting in Hadoop on start-dfs.cmd
So I am trying to setup Hadoop by using this as reference: Towardsdatascience.com click here
now the error:
E:\RIYA\hadoop-env\hadoop-3.2.1\sbin>start-dfs.cmd
2 cmd pop up, one for datanode and one for namenode
DataNode…
Riya Pal
- 55
- 1
- 5
0
votes
0 answers
Spark Executors are running on node manager machine in spite machine is in decommission state
we have a spark cluster of 50 nodes, with YARN as the resource manager.
cluster is based on HDP version - 2.6.4 and , and cluster is based ambari platform
on yarn node manager yarn-machine34 , we set the machine as decommission state and also for…
Judy
- 1,595
- 6
- 19
- 41
0
votes
1 answer
Whenever I restart my ubuntu system (Vbox) and start my hadoop , my name node is not working
Whenever I restart my ubuntu system (Vbox) and start my Hadoop, my name node is not working.
To resolve this I have to always the folders of namenode and datanode and format Hadoop every time I restart my system.
Since 2 days am trying to resolve…
Gaurav
- 1
0
votes
1 answer
ERROR datanode.DataNode: Exception in secureMain
I was trying to install Hadoop on windows.
Namenode is working fine but Data Node is not working fine. Following error is being displayed again and again even after trying for several times.
Following Error is being shown on CMD regarding dataNode:
…
Muhammad Alim
- 1
- 1
0
votes
0 answers
no datanodes created when running "jps" command
I'm new at Hadoop and when I run "jps" command shows that no data node exist but when I run stop-all.sh or start-all.sh it shows that there is a data node started and stopped how can I solve this issue
I tried so many solutions but no one works also…
moazmahmoud saad
- 1
- 1
- 3
0
votes
1 answer
hdfs put fails from laptop to remote hadoop cluster
I have my hadoop cluster set up on a different network. Because of this, hdfs put is failing when I run it from my laptop.
Is there a port I should forward or something to access the datanodes remotely? I see it's using the local ip address in the…
steven hurwitt
- 183
- 2
- 15
0
votes
1 answer
Datanode not working on Hadoop single node cluster on windows
There are many similar questions on stack overflow but none of them solves my problem.
I'm trying to start my namenode and datanode, of which namenode starts working but datanode fails alongwith resource manager and node manager. Here is the error…
sheharbano
- 211
- 2
- 13
0
votes
1 answer
If the number of hadoop data node folders is different, can block replication problems occur?
Suppose you have 20 nodes in cluster, 15 nodes have 10 child folders,
like /data01, ... /data10 and other 5 nodes have 6 child folders. The number of data node folder is unbalanced.
In this case, is the possibility of creating a problem in the block…
user11589767
0
votes
0 answers
ambari + how to disable new datanode machine from data replication
we have ambari and HDP 2.6.4 cluster with 245 workers machines
each worker have ( datanode component and resource manager component )
now we want to add 10 new workers machines to the cluster
but we want to disable the datanode machines so no HDFS…
Judy
- 1,595
- 6
- 19
- 41
0
votes
0 answers
HDFS Dead Datanode
I'm working in a HDP-3.1.0.0 environment, the HDFS version I'm using is the 3.1.1.3.1, the cluster is composed by 2 Namenodes and 4 Datanodes.
After a reboot of the HDP services (stop all and start all), the cluster seems working well, but I see the…
GiuVi
- 23
- 4
0
votes
1 answer
HDFS replica + and min data nodes number in the HDFS cluster
we have Hadoop cluster with only 2 data nodes machines
in HDFS configuration we defined the Block replication to 3
so
Block replication=3
is it OK? to defined Block replication=3 , when we have only two data nodes in the cluster
from my…
Judy
- 1,595
- 6
- 19
- 41
0
votes
1 answer
hadoop cluster + any way to disable spark application to run on specific data nodes
we have Hadoop cluster ( HDP 2.6.5 cluster with ambari , with 25 datanodes machines )
we are using spark streaming application (spark 2.1 run over Hortonworks 2.6.x )
the current situation is that spark streaming applications runs on all datanodes…
jessica
- 2,426
- 24
- 66
0
votes
1 answer
Failure to start Hadoop after having stopped a running (and working) instance before, because Datanode says that the directory is locked
I have a cluster running Hadoop 1.2.1 with Giraph on top. The server runs ok, but when I stop it, I am unable to make it run again. In the datanode log I get the following error: ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:…
ricks1337
- 1
- 1
0
votes
0 answers
HDFS not accessible from a java application
I have a clouder quickstart container and I try to get the list of file under HDFS using a java application;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import…
C.yasser
- 11
- 7
0
votes
1 answer
Wrong IP mapping on some data nodes in hadoop
I have a hadoop setup on 7 nodes configured using local domains using /etc/hosts.
It looks like this
1.2.3.4 hadoop-master
1.2.3.5 hadoop-slave-1
1.2.3.6 hadoop-slave-2
1.2.3.7 hadoop-slave-3
1.2.3.8 hadoop-slave-4
1.2.3.9 hadoop-slave-5
1.2.3.10…
Avik Aggarwal
- 599
- 7
- 28