1

Environment: The production cluster has 2 name-nodes (active and standby namely) and the nodes are SAS drives in Raid-1 configuration. These nodes have nothing but the master services (NN and Standby NN) running on each. They have a Ram of 256GB while the data nodes (where most of the processing happens) are set with only 128GB.

My Question: Why does Hadoop’s master nodes have such high Ram and why not the Datanodes when most of the processing is done where the data is available.?

P.S. As per hadoop thumb rule, we only require 1GB for every 1million files.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Santosh Bitra
  • 61
  • 4

1 Answers1

0

The Namenode stores all file references from all the datanodes in memory.

The datanode process doesn't need much memory, it's the YARN NodeManagers that do

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245