Stack Overflow
Stack Overflow

Questions tagged [namenode]

The Hadoop NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept.

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself.

Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The NameNode responds to the successful requests by returning a list of relevant DataNodes where the data lives.

200 questions
2
votes
2 answers

How to split the network load in hadoop HDFS

I have 2 servers hadoop one is the namenode and the other the secondary namenode. both are datanode and currently when I want to read a file using the port of the namenode 8020 it works, but all the network load goes to that node, is there no way to…
jerson
  • 345
  • 4
  • 10
2
votes
1 answer

Namenode Shutdown Error - Exiting with Status 0 (Hadoop Installation)

I am trying to get Hadoop-2.8.1 working. I am running the command to configure the Namenode. However, Namenode shuts down when I run it from the Hadoop directory. ***********s-MacBook-Pro-2:~ ***********$ cd…
user1072337
  • 12,615
  • 37
  • 116
  • 195
2
votes
2 answers

Can anyone say reasons for not showing hadoop demons in jps?

I am using hadoop 2.7.3 version and after configuration namenode is not showing in jps. can anyone say what is the reason i gave correct permissions to the concerned files. I have deleted /tmp files and recreated and after reformatting the namenode…
Yasodhara
  • 111
  • 2
  • 13
2
votes
1 answer

Do we need to put namenode in safe mode before restarting the job tracker?

I have a Hadoop cluster running Cloudera's CDH3, Apache Hadoop's 0.20.2 equivalent. I want to restart the job-tracker as there are some jobs which are not getting killed. I tried killing them from the command line, the command executes successfully,…
Varun Risbud
  • 143
  • 9
2
votes
1 answer

Call From kv.local/172.20.12.168 to localhost:8020 failed on connection exception, when using tera gen

I am working with hadoop teragen to check the hadoop mapreduce benchmarking with the terasort. But when i run the following command, hadoop jar /Users/**/Documents/hadoop-2.6.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar teragen…
KayV
  • 12,987
  • 11
  • 98
  • 148
2
votes
1 answer

Hadoop: Why I got "Max Non Heap Memory is -1 B." message in Namenode Information Web? What it does mean?

I have a cluster to use Hadoop 2.6.5 (one master that works as namenode and datanode, and two slaves) that I made using VirtualBox (easch node has Xubuntu 16.04 installed). A priori, the installation is right because I ran a wordcount example and…
mjbsgll
  • 722
  • 9
  • 24
1
vote
1 answer

writing to hdfs error: Not enough replicas was chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1}

I encountered the problem similar to this: https://stackoverflow.com/questions/52809233/failed-to-place-enough-replicas-expected-size-is-1-but-only-0-storage-types-can my name node logs in $HADOOP_HOME/logs/ while the storage policy is set to…
mehran
  • 39
  • 3
1
vote
1 answer

How to access Namenode UI with KnoxSSO

My goal I want to access Namenode UI through KnoxSSO and browse hdfs files without Ambari. What have I already done I've already set up Security cluster and HDFS HA with QJM. Apache Knox uses Pac4J provider to authenticate users with Openid…
venom13k
  • 53
  • 4
1
vote
0 answers

ERROR: Cannot set priority of secondarynamenode process 84665

I recently installed Hadoop on my macbook pro with m1 chip using homebrew. After setting up some necessary configurations, when I tried to start-dfs.sh I got this log: ╰─ start-dfs.sh …
david
  • 11
  • 2
1
vote
0 answers

HDFS: namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(390)) - Error: flush failed for required journal

On one of our platforms, HDFS namenode is shutting down with following error message every 1 or 3 days FATAL namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(390)) - Error: flush failed for required journal (JournalAndStream(mgr=QJM to…
dsoguet
  • 11
  • 1
1
vote
1 answer

How often are blocks on HDFS replicated?

I have a question regarding hadoop hdfs blocks replication. Suppose a block is written on a datanode and the DFS has a replication factor 3, how long does it take for the namenode to replicate this block on other datanodes? Is it instantaneuos? If…
wayne
  • 11
  • 2
1
vote
0 answers

What is the prefered solution for corrupted namenode metadata

we have HDP cluster , version 2.6.5 cluster include management of two name-node ( one is active and the secondary is standby ) and 65 datanode machines we have problem with the standby name-node that not started and from the namenode logs we can see…
jessica
  • 2,426
  • 24
  • 66
1
vote
0 answers

Add datanode and backup existed data to a standalone Hadooop on windows machine

I have installed a standalone mode Hadoop on windows machine locally, with one datanode and the replication factor set as 1. I have already uploaded some data onto the datanode. Let us call this existing datanode as datanode1. I would like to add…
XYZ
  • 352
  • 5
  • 19
1
vote
0 answers

ambari + namenode HA and hdfs zkfc -formatZK

we have production HDP cluster , HDP version - 2.6.5 Ambari version - 2.6.1 We are performing now the change hostname configuration according to the document -…
Judy
  • 1,595
  • 6
  • 19
  • 41
1
vote
0 answers

Airflow conn_id with multiple server

I am using WebHDFSSensor and for that we need to provide namenode. However, active namenode and standBy namenode change. I can't just provide current namenode host to webhdfs_conn_id. I have to create connection from both host. I tried to provide…
Ayush Goyal
  • 415
  • 4
  • 23
1 2
3
…
13 14