0

We are having 20 data nodes and 3 management nodes. Each data node is having 45GB of RAM . 

Data node RAM Capacity
45x20=900GB total ram 

Management nodes RAM Capacity
100GB x 3 = 300GB RAM

I can see memory is completely occupied in Hadoop resource manager URL and Jobs submitted are in waiting state since 900GB is occupied till 890GB in resource manager url.

However , I have raised a request to Increase my memory capacity to avoid memory is being used till 890Gb out of 900GB.

Now, Unix Team guys are saying in data node out of 45GB RAM 80% is completely free using free -g command (cache/buffer) shows the output as free . However in Hadoop side(resource manager) URL says it is completely occupied and few jobs are in hold since memory is completely occupied.I would like to know how hadoop is calculating the memory in resource manager and is it good to upgrade the memory since it is occupying every user submit a hive jobs .

Who is right here hadoop output in RM or Unix free command .

Ananya
  • 1
  • 1
  • This Q is not about programming as defined for StackOverflow. It **may** be more appropriate on thttp://unix.stackexchange.com OR http://serverfault.com . Use the `flag` link at the bottom of your Q and ask the moderator to move it. Please don't post the same Q on 2 different sites. Please read https://stackoverflow.com/help/on-topic , http://stackoverflow.com/help/how-to-ask , http://stackoverflow.com/help/dont-ask and http://stackoverflow.com/help/mcve before posting more Qs here. Good luck. – shellter Oct 20 '17 at 04:26

1 Answers1

0

The UNIX command free is correct because the RM shows reserved memory not memory used.

If I submit a MapReduce job with 1 map task requesting 10GB of memory per map task but the map task only uses 2GB then the system will only show 2GB used. The RM will show 10GB used because it has to reserve that amount for the task even if the task doesn't use all the memory.

tk421
  • 5,775
  • 6
  • 23
  • 34
  • However , That reserved memory can't use by other map reduce tasks unless until the job gets completed?. As of now when we submit new job and it says as Pending state since complete memory is occupied . Need to understand the whole flow . – Ananya Oct 20 '17 at 12:24
  • Exactly. The RM has to assume the resources requested (memory) are in use. Just so you know, there is a lot of tuning that needs to be done to YARN to get the behavior you want. – tk421 Oct 20 '17 at 17:46
  • Out of 45GB --> Node Manager is having 41GB in Resource manager URL . – Ananya Oct 22 '17 at 21:27
  • 4GB is reasonable. A worker node running the Node Manager + the Data Node needs memory too. – tk421 Oct 23 '17 at 17:42