0

I have a master node and a slave node in hadoop cluster.

Master is also configured to function as slave by starting nodemanager & datanode in it.

Map phase is working in both master and slave node perfectly.

I have only 1 reduce phase, but it runs only in slave node.

Slave node has less disk capacity so when it fails reduce task will work in master node.

I do not want to start reduce task in slave node.

Is there any configuration to prevent reduce task to invoke in a specific node (in my case in slave node)?

In master all hadoop services are running. In slave nodemanager & datanode are running.

Eugene Lisitsky
  • 12,113
  • 5
  • 38
  • 59
Ram
  • 11
  • 2
  • Do you have only these two nodes in the cluster? From which node are you starting the MR job? From the master node or slave node or is there any other client machine in or outside the cluster from where you are submitting the job? – Manoj Kumar G Dec 17 '16 at 08:45
  • only 2 nodes. from master node i am starting the MR job. – Ram Dec 17 '16 at 09:03
  • i just want only 1 reducer which is to be in master node, because in slave node not much disk space for reducer to complete. now only after the termination of the reducer container in slave node (due to lack of disk space) the reducer task starts in master. – Ram Dec 17 '16 at 09:15
  • I dont think it is possible. By design, hadoop automatically takes care of allocating the resources to map and reduce tasks. If in case, you have more nodes, hadoop chooses the nodes near to it from the topology and runs tasks on them. Since you have only two nodes, this is not possible in your case. However, if your master has more resources (memory and cpu), hadoop should consider launching the tasks on the Master node itself. Try allocating more resources on Master node. – Manoj Kumar G Dec 21 '16 at 05:24
  • I will try allocating more resources. Thank You – Ram Dec 21 '16 at 14:27

0 Answers0