Questions tagged [namenode]

The Hadoop NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept.

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself.

Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The NameNode responds to the successful requests by returning a list of relevant DataNodes where the data lives.

200 questions
0
votes
1 answer

What is the meaning of DSQUOTA and NSQUOTA columns in Hadoop FsImage?

I have taken a dump of fsimage using OIV tool and NSQUOTA and DSQUOTA fields always have one of the following values NSQUOTA 1 null -1 0 DSQUOTA -1 0 What does NSQUOTA and DSQUOTA represent? How to interpret these values?
Aravind Yarram
  • 78,777
  • 46
  • 231
  • 327
0
votes
0 answers

NameNode memory consumption is much larger than expected

According to some articles,namenode memory consumption would be like: "Each file or directory or block occupies about 150 bytes in the namenode memory." I found my cluster have 172,963,375 files and directories, 187,792,367 blocks = 360,755,742…
0
votes
3 answers

Hadoop 3.1.1 Mac OS Namenode Issues

I had a Hadoop on my machine running but I was running into some compiler issues, so I deleted it and started fresh. I was following this setup: https://www.guru99.com/how-to-install-hadoop.html When I run $HADOOP_HOME/bin/hdfs namenode…
Niyya Té
  • 71
  • 1
  • 1
  • 7
0
votes
3 answers

How to track HDFS Active Namenode change event in NiFi?

I have HDFS cluster with Active and Stanby Namenodes. Sometimes when cluster gets restarted, Namenodes exchange their roles - Standby becomes Active, and vice versa. Then I have NiFi flow with PutParquet processor writing some files to this HDFS…
megazlo
  • 3
  • 2
0
votes
1 answer

Hadoop : swap DataNode & NameNode without losing any HDFS data

I have a cluster of 5 machines: 1 big NameNode 4 standard DataNodes I want to change my current NameNode with a DataNode without losing the data stored in HDFS, so my cluster could become: 1 standard NameNode 3 standard DataNodes 1 big…
Noa Be
  • 277
  • 3
  • 10
0
votes
1 answer

How to restore data from BP folder in HDFS

Configuration : We have 1 HDFS Namenode & 3 data nodes. Now, namenode got restarted and for some reason cluster ID got changed (Datanodes were still running) We tried updating Cluster ID from datanode to namenode and vice-versa but no luck.…
Sunil Agarwal
  • 4,097
  • 5
  • 44
  • 80
0
votes
0 answers

Active Namenode is Down in Hadoop

In my hadoop cluster i have 1 active name node 1 standby name node 3 journal nodes 4 data nodes Upto my analysis, Active Namenode in down, because it can't able to write editlogs to majority of journal node. standby name node did't take over after…
0
votes
2 answers

How to get the memory usage metrices from hadoop ResourceManager

I have a dataproc cluster running in Google cloud with 3 master nodes and 40 worker nodes. I use hadoop ResourceManager to monitor the applications and other metrics like queue usage, memory usage, app running etc. I am unable to get the command to…
0
votes
0 answers

Increase efficiency of name and data node in HDFS

i have 1 name node and 2 data node with 32 gb ram in HDFS. But it is not taking more than 30 people at a time.Is there a way to improve the efficiency of name and data node
Didhin
  • 1
  • 1
0
votes
3 answers

Failed to load an FSImage file! || How to solve

I am trying to show all the services using the Jps command, but when i hit the console the below nodes are only showing 3633 SecondaryNameNode 4228 Jps 3493 DataNode 4198 NodeManager 4088 ResourceManager I am trying to start all services using…
Mandrek
  • 1,159
  • 6
  • 25
  • 55
0
votes
0 answers

Shutting down NameNode at ubuntu/127.0.1.1

my system composed of : Ubuntu x32 bit (i386.iso) hadoop-1.2.1 java 1.8.0_171 so when i run this command : hadoop namenode -format i get those errors : WARN util.NativeCodeLoader: Unable to load native-hadoop library for your…
Mehdi Hadji
  • 105
  • 2
  • 10
0
votes
2 answers

Unable to start Hadoop (3.1.0) in Pseudomode on Ubuntu (16.04)

I am trying to follow the Getting Started guide from the Hadoop Apache website, in particular from the Pseudo distributed configuration, Getting started guide from Apache Hadoop 3.1.0 but I am unable to start the Hadoop Name- and Data Nodes. Can…
robbie70
  • 1,515
  • 4
  • 21
  • 30
0
votes
0 answers

Issues with Hadoop upgrade from hadoop 1.1.2 to hadoop 2.9.0

I am trying to upgrade Hadoop from 1.1.2 to 2.9.0 I had executed the command start-dfs.sh -upgrade on hadoop 2.9.0 cluster. It shows that upgrade is in process and not yet finalized. I have been waiting for long but still showing the same status.…
0
votes
1 answer

Everything else starting on Hadoop pseudo-distributed except namenode

I have Hadoop 2.9.0 on Ubuntu at /usr/local/hadoop But when I try start-dfs.sh No error is shown while starting namenode But when I type jps, only 10900 SecondaryNameNode 11047 Jps 10696 DataNode Seams to have started, not namenode Things…
0
votes
0 answers

hadoop live datanodes are keep changing

I am facing a very strange issue. I have installed the HDP 2.6.4 cluster using ambari on openshift containers. Everything working fine, except one thing, I have added the 8 datanodes and out of these only 5 datanodes are live at a time. Main issue…
Amit Bondwal
  • 133
  • 1
  • 2
  • 12