Stack Overflow
Stack Overflow

Questions tagged [datanode]

A DataNode stores data in the HFS (HadoopFileSystem). A functional filesystem has more than one DataNode, with data replicated across them.

86 questions
0
votes
1 answer

Datanode is not starting. Namenode , secondary name node everything starts normally

20/01/20 22:18:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/01/20 22:18:12 INFO checker.ThrottledAsyncChecker: Scheduling a check for…
Chaitra
  • 9
  • 1
  • 6
0
votes
1 answer

In Hadoop 3.1.0 namenode is working but datanode is not working

In Hadoop 3.1.0 namenode is working but datanode is not working showing below message: STARTUP_MSG: build = https://github.com/apache/hadoop -r 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d; compiled by 'centos' on 2018-03-30T00:00Z STARTUP_MSG: java…
Abinas Sahu
  • 1
0
votes
1 answer

How does a datanode know next datanode in the writting pipeline?

From "White, Tom; Hadoop: The Definite Guide; Ch. 3, The Hadoop Distributed Filesystem, Anatomy of a File Write": ...The DataStreamer streams the packets to the first datanode in the pipeline, which stores each packet and forwards it to the second…
Dipperman
  • 119
  • 1
  • 12
0
votes
1 answer

Deleting HDFS Block Pool

I am running a Spark on Hadoop cluster. I tried running a Spark job and noticed I was getting some issues, eventually realised by looking at the logs of the data node that the file system of one of the datanodes is full I looked at hdfs dfsadmin…
soundofsilence
  • 330
  • 2
  • 12
0
votes
1 answer

Secure Hadoop - Start Datanode in background

I'm running hadoop 3.1.2 and I'm setting authentication on my cluster. For performance reasons, I do not want data encryption on my setup, only authentication. I've read that part about securing datanodes…
Ror
  • 141
  • 2
  • 10
0
votes
0 answers

datanode took long time to become live after restart namenode or reboot

As title, it always took about 20 mins or more for datanode to become live in hdfs summary. I couldn't find any clue about it. Is there any way to speed up the process ? Thanks.
Chandler.Huang
  • 873
  • 3
  • 12
  • 24
0
votes
1 answer

All the slaves in the Hadoop cluster should be of the same configuration

In Hadoop is it mandatory that all the slaves in the Hadoop cluster should be of the same configuration?
Sands
  • 11
0
votes
0 answers

Cannot create directory in HDFS. Name node is in safe mode

I'm have deployed Hadoop in docker which is running on aws ec2 ubuntu AMI instance. when I try to create a directory in hdfs it says Cannot create directory. Name node is in safe mode below are the properties in hdfs-site.xml name:…
richie
  • 13
  • 5
0
votes
1 answer

Datanode disks are full because huge files as stdout

we have the follwing hadoop cluster versions , ( DATA-NODE machine are on Linux OS version - 7.2 ) ambari - 2.6.1 HDP - 2.6.4 we saw few scenarios that disks on datanode machine became full 100% and that because the files as - stdout are huge…
Judy
  • 1,595
  • 6
  • 19
  • 41
0
votes
1 answer

Datanode + Error occurred during initialization of VM Too small initial heap

we restart the dastanodes on our cluster we have 15 Data node machines in the ambari cluster while each datanode machine have 128G RAM versions - ( HDP - 2.6.4 and ambari version 2.6.1 ) but datanode failed to start on the follwing error Error…
Judy
  • 1,595
  • 6
  • 19
  • 41
0
votes
1 answer

Hadoop 3.1.1 showing only one live datanode

I installed hadoop 3.1.1 on 3 virtual machines with VMware on Ubuntu. When I run hdfs namenode -format and start-all.sh then jps works correctly on my master and two slaves nodes. However, with the command hdfs dfsadmin -report, I can see only one…
Juan
  • 184
  • 1
  • 4
  • 16
0
votes
1 answer

can not start the hadoop datanode hadoop 3.1.1 in ubuntu 18.04

I recently install the hadoop 3.1.1 to my computer in single cluster mode.after do all the configuration for hdfs-site.xml,yarn-site.xml,core-site.xml,mapred-site.xml and hadoop-env.sh following commands work without any…
Mihiran Chathuranga
  • 116
  • 2
  • 15
0
votes
1 answer

Data-node Alive from ambari isnt stable

we have ambari cluster version - 2.6.1 and hadoop version 2.6.4 number of datanode are - 10 from ambari dashboard we can see the window that show the follwing: DataNodes Live 9/10 but after few min all datanode are alive as: DataNodes…
Judy
  • 1,595
  • 6
  • 19
  • 41
0
votes
1 answer

hadoop - Multiple datanode configuration in Pseudo-distributed mode

I am newbie in hadoop. I have setup hadoop - Pseudo-distributed mode in single machine. My hdfs-site.xml configuration as default: dfs.replication 1 …
Huy Truong
  • 25
  • 1
  • 6
0
votes
2 answers

How can we exclude some data nodes only from one namespace in hdfs federation?

I was checking the hadoop federation. Here the data nodes are common, only namespace will be different. How can we exclude some datanodes only from one namespace.
Rahul
  • 459
  • 2
  • 13
1 2 3 4
5
6