Stack Overflow
Stack Overflow

Questions tagged [datanode]

A DataNode stores data in the HFS (HadoopFileSystem). A functional filesystem has more than one DataNode, with data replicated across them.

86 questions
1
vote
0 answers

A file bigger than Hadoop cluster storage

What would happen if a file I store in HDFS is 5GB but there are only 3 DataNodes 1GB each? Lets say I store a file 3GB in HDFS having 4 DataNodes 1GB each. After processing I have some results.txt. What will happen to processed file blocks which…
SliceOfPig
  • 123
  • 3
  • 10
0
votes
0 answers

a decommissioned datanode become normal again

A datanode in HDFS was decommissioned by putting its hostname in /etc/hadoop/excludeHosts and excuting hdfs dfsadmin -refreshNodes. Then I did nothing. I made sure it was decommissioned by exeuting "hdfs dfsadmin -report". Several hours later. I…
LeeDerson
  • 47
  • 1
  • 5
0
votes
1 answer

How to move data block from datanode to other datanode during mapreduce?

I implemented 4 node cluster for running hadoop following the site(https://www.linode.com/docs/guides/how-to-install-and-set-up-hadoop-cluster/) By the way, I want to move data block to other datanode after map task or during map or reduce task. Is…
arthur seokwon choi
  • 11
  • 1
0
votes
1 answer

How to start data node(s) and compute node(s) without accessing the DolphinDB web interface?

I have executed sh startController.sh and sh startAgent.sh to start the controller(s) and agent(s). But I cannot start data node(s) and compute node(s) via the web interface, because the server port is not open. Does anyone have best practices about…
Claire
  • 238
  • 4
0
votes
1 answer

How to have a cluster of 3 datanodes that work at the same time?

I run one datanode with: ./bin/hdfs datanode -conf ./etc/hadoop/datanode1.xml only one work when i try run two: "datanode is running as process. Stop it first and ensure /tmp/hadoop-user-datanode.pid file is empty before…
Pypthon3
  • 1
  • 1
0
votes
1 answer

HDFS unreachable outside of network

How do I properly reach my hdfs in the cloud from my laptop with hdfs dfs command? I set up a HDFS on EC2 following this guide and I've set up my (AWS) security group to allow all inbound and outbound traffic on my Network interface (so I'm ruling…
Jenny Lian
  • 79
  • 4
0
votes
0 answers

Why cant I change datanode port in HDFS

I read https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml and want my datanode to run on port 9300. My https-site.xml is as follows: