Stack Overflow
Stack Overflow

Questions tagged [namenode]

The Hadoop NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept.

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself.

Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The NameNode responds to the successful requests by returning a list of relevant DataNodes where the data lives.

200 questions
1
vote
1 answer

Metadata storage by Namenode

The namenode also knows the datanodes on which all the blocks for a given file are located, however, it does not store block locations persistently, since this information is reconstructed from datanodes when the system starts. What does…
pareekrachit
  • 15
  • 1
  • 3
1
vote
2 answers

Error While running Namenode

While i am starting namenode i am getting the below error: [sudo] password for hadoopnode1: * Starting Hadoop namenode: starting namenode, logging to /var/local/hadoop/logs/hadoop-hdfs-namenode- hadoopnode1.vertexcs.com.out log4j:ERROR Could not…
ravi
  • 43
  • 1
  • 5
0
votes
2 answers

hdfs reformat name node failed by error Unable to check if JNs are ready for formatting

I want to start fresh hdfs cluster, I start zookeeper and yarn with command : ./zkServer.sh start && start-yarn.sh after above command I started journalnode on each 5 node with command: hdfs --daemon start journalnode now, before starting namenodes…
Amir
  • 1
  • 1
0
votes
1 answer

How can I resolve the missing blocks inconsistency issue between 2 name nodes

After performing decommission and recommission tasks, I encountered an issue with missing blocks, resulting in an inconsistency problem. In Ambari, I received a "Namenode Health" alert with the following details: nn01: Total Blocks: 3,607,770,…
krrng
  • 1
0
votes
0 answers

Namenode is not showing in JPS

Namenode is not working When I run the command start-all.sh it says running namenode[localhost] But when I run JPS command namenode is not showing. Everytime I have to run hadoop namenode -format in order to run namenode,is their any other solution?
Ravula Ramakrishna
  • 1
0
votes
1 answer

How to have a cluster of 3 datanodes that work at the same time?

I run one datanode with: ./bin/hdfs datanode -conf ./etc/hadoop/datanode1.xml only one work when i try run two: "datanode is running as process. Stop it first and ensure /tmp/hadoop-user-datanode.pid file is empty before…
Pypthon3
  • 1
  • 1
0
votes
1 answer

HDFS unreachable outside of network

How do I properly reach my hdfs in the cloud from my laptop with hdfs dfs command? I set up a HDFS on EC2 following this guide and I've set up my (AWS) security group to allow all inbound and outbound traffic on my Network interface (so I'm ruling…
Jenny Lian
  • 79
  • 4
0
votes
1 answer

How can I solve this problem : hadoop namenode -format gives error?

I setup Hadoop HA Cluster. I try to format hadoop as below code: hdfs --loglevel DEBUG namenode -format -force then when I see the log ( it is not DEBUG mode and this is another problem which i ve not solved) it gives error as below: error…
CompEng
  • 7,161
  • 16
  • 68
  • 122
0
votes
0 answers

HA NameNode exits unexpected with Error replaying edit log at offset. Expected transaction ID was xxxx

2 HA Namenode exits at the same time, can't restart after. The error log is: org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 95968824. Expected transaction ID was 2615482563 Recent opcode offsets:…
dailybird
  • 23
  • 4
0
votes
1 answer

NameNode Format error "failure to login for principal: X from keytab Y: Unable to obtain password from user" with Kerberos in a Hadoop cluster

I've been setting up Kerberos with my Hadoop cluster on Ubuntu 20.04.1 LTS and when I try to reformat the namenode in command line after changing all config files and setting everything up (including principals and keytabs), I'm being met by the…
MaxRS7_
  • 3
  • 2
0
votes
1 answer

"ERROR: Cannot set priority of secondarynamenode process 31231"

I have a problem with Hadoop. I am on mac OS and I have a problem when I want to launch my node. I installed Hadoop this way : brew install hadoop I also configured the different files like this : hadoop-env.sh : export HADOOP_OPTS="$HADOOP_OPTS…
Junot
  • 137
  • 6
0
votes
0 answers

How to tune the CMSInitiatingOccupancyFraction in Hadoop cluster

We have an old Hadoop cluster based on HDP from Hortonworks version HDP 2.6.4. The cluster includes 2 namenode services where one is the standby namenode and the second is the active namenode. All machines in the cluster are RHEL 7.2 version, and we…
Judy
  • 1,595
  • 6
  • 19
  • 41
0
votes
0 answers

Getting IP adress of a NameNode in HDFS

I want to obtain the IP adress of an HDFS namenode. like for example !hdfs getconf -confkey -nnRpcAddresses But the output is the URL with the ID. Is there something similar but with the IP as an output?
osled
  • 1
  • 1
0
votes
0 answers

I can't get the NameNode to start in Hadoop

I'm trying to run Hadoop on a single-node in a pseudo-distributed mode. I'm using Ubuntu 20.04 on WSL and have Java 8. When I run: start-dfs.sh start-yarn.sh And then run: jps My system outputs the following: 1829 SecondaryNameNode 2549 Jps 1612…
matt24ck
  • 11
  • 1
0
votes
1 answer

Does the memory requirements for secondary Namenode is in the same range as that of Namenode? in hadoop

I am working with hadoop, little bit curious about the memory requirements for secondary namenode... as compared with Namenode
Atharv Chaudhari
  • 13
  • 4
1 2 3
…
13 14