Stack Overflow
Stack Overflow

Questions tagged [namenode]

The Hadoop NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept.

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself.

Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The NameNode responds to the successful requests by returning a list of relevant DataNodes where the data lives.

200 questions
3
votes
1 answer

hadoop cluster with active standby namenode + gap in the edit log

we have ambari cluster , HDP version 2.6.5 cluster include management of two name-node ( one is active and the secondary is standby ) and 65 datanode machines we have problem with the standby name-node that not started and from the namenode logs we…
jessica
  • 2,426
  • 24
  • 66
3
votes
0 answers

NameNode only has to wait for blocks to be minimally replicated before returning successfully

I have a question about the block reports sent to the NameNode from the DataNodes when a client writes in HDFS, as well as the ack from the NameNode to the client about closing the file. Could someone kindly elaborate this paragraph of the Hadoop…
triples
  • 31
  • 2
3
votes
1 answer

Where would namenode and datanode be installed if not defined in hdfs-site.xml?

My hdfs-site.xml has ONLY the following: dfs.replication 1 Question. Where would the NameNode and DataNode be installed? I am using…
Gautam De
  • 39
  • 4
3
votes
0 answers

Java Native Memory Internal Section

Running a Hadoop namenode proc on JDK 1.8.0.92 with Xmx and Xmx = 75Gb we observed a spike in system memory within x interval. And slowly an OOM happened. During the debugging process we found the following: RSS went higher than Xmx Is that heap -…
Amith sha
  • 31
  • 2
2
votes
1 answer

Why Impala spend a lot of time Opening HDFS File (TotalRawHdfsOpenFileTime)?

I find that my Impala swarm performs not stable, normally it takes only a few seconds (less than 10s) to finish a query, but occasionally it will take more than 40s (and this situation will last for a few minutes), and when that happens, accroding…
luochen1990
  • 3,689
  • 1
  • 22
  • 37
2
votes
6 answers

Hadoop namenode metadata

I am a bit confused by the Hadoop architecture. What kind of file metadata is stored in Hadoop Namenode? From Hadoop wiki, it says Namenode stores the entire system namespace. Does information like last modified time, created time, file size,…
leon
  • 10,085
  • 19
  • 60
  • 77
2
votes
1 answer

Hadoop port 50070 in use Windows installation error

I am a newbie in hadoop and trying to create a standalone hadoop cluster in windows. I am getting an error while starting the name node as shown below. However, upon checking in windows for the process using the port 50070 i could not find any…
Vincent Kr
  • 21
  • 1
2
votes
0 answers

HDFS NameNode startup very slow with few blocks

I have a fairly small setup (HDP 2.6) with roughly 1429 blocks on a 15 TB HDD. The system has 512 GB RAM and 128 cores (256 threads). Over last few days, I've seen the startup of entire HDP setup go from about 10 minutes to several hours. The…
MojoJojo
  • 3,897
  • 4
  • 28
  • 54
2
votes
0 answers

Connecting to HDFS namenode running in docker container from outside host VM

I have a HBase + HDFS setup, in which each of the HBase master, regionservers, HDFS namenode and datanodes are containerized. When running all of these containers on a single host VM, things work fine as I can use the docker container names…
anthr
  • 1,026
  • 4
  • 17
  • 34
2
votes
2 answers

Hadoop 3.1.2 Failed to retrieve data from /webhdfs/v1/?op=LISTSTATUS: Server Error

I'm using hadoop 3.1.2 and jdk-12.0.1 operating on Pseudo-Distributed mode. So far from the command line every thing seems to be working normal I can upload files retrieve from hdfs and local. For some reason when I open the namenode which is…
user3719458
  • 346
  • 3
  • 12
2
votes
1 answer

Do we have to format datanode for HDFS file system or we only need to format namenode

While setting up pseudo or full cluster, do we need to format data node to install HDFS file system or we only have to format namenode. I am asking this because, I have read this on many blogs that, HDFS stores the data in sequential order on the…
Sagar Chawla
  • 59
  • 3
2
votes
3 answers

Hadoop start-all.cmd command : datanode shutting down

I am trying to install hadoop in my windows 10 Reference : https://github.com/MuhammadBilalYar/Hadoop-On-Window/wiki/Step-by-step-Hadoop-2.8.0-installation-on-Window-10 Hadoop start-all.cmd command start namenode , resourceManager and nodeManager…
Bharti Ladumor
  • 1,624
  • 1
  • 10
  • 17
2
votes
0 answers

How to handle authentication for Hadoop web UI?

In my hadoop cluster . we had to authenticate the web UI components so started to search and find this option https://streever.atlassian.net/wiki/spaces/HADOOP/blog/2014/03/07/491558/Securing+Hadoop+HDP+Web+UI+Component+s I had used the…
Albin Paul
  • 3,330
  • 2
  • 14
  • 30
2
votes
1 answer

Hadoop Single Node Cluster setup error during namenode format

I have installed Apache Hadoop 2.6.0 in Windows 10. I have been trying to fix this issue but failed to understand the error or any mistake from my end. I have set up all the paths correctly, Hadoop version is showing the version in command prompt…
Bhaskar Das
  • 652
  • 1
  • 9
  • 28
2
votes
3 answers

Hadoop namenode not starting with exit code 1

prateek@prateek:~$ start-dfs.sh Starting namenodes on [localhost] pdsh@prateek: localhost: ssh exited with exit code 1 Starting datanodes Starting secondary namenodes [prateek] prateek@prateek:~$ jps 11011 SecondaryNameNode 10787 DataNode 11161…
Prateek Tater
  • 21
  • 1
  • 2
1
2
3
…
13 14