Suppose we have metadata generated for years and fsimage have data more than RAM memory. Now when namenode starts it loads fsimage in-memory but in such case where fsimage size itself is more than RAM then how hadoop handles it?
Asked
Active
Viewed 106 times
0
-
If there's not enough RAM, you should expect an OOM exception – OneCricketeer Aug 01 '22 at 13:22
-
Thanks for the reply. How to handle such case? – nayak0765 Aug 01 '22 at 14:27
-
You'd need to provision a machine with more RAM or find a way to reduce the fsimage size, potentially without a running namenode – OneCricketeer Aug 01 '22 at 17:50
-
I am new to hadoop. correct me if i am wrong.My understanding is we can have commodity servers for hadoop setup. Suppose we have stored petabytes of data in cluster.Now if fsimage grows in size in namenode server and has huge metadata . Then for namenode we would need to have big server with high RAM configuration ,so that namenode can load fsimage file in-memory. So how can we manage namenode server with commodity machine. – nayak0765 Aug 03 '22 at 05:29
-
"Commodity" doesn't imply "small". Production Hadoop clusters I've worked with had 64-128GB of RAM, which is easy to get in Cloud resources. And that was tracking over 5PB of data. If your fsimage gets that large, then you have other problems such as too many small files in the cluster. There's no direct correlation between datanode sizes and fsimage sizes. If you store 100 1GB files, it'll have a smaller fsimage than 100000 1MB files – OneCricketeer Aug 03 '22 at 14:19