3

I'm new to Hadoop and I want to limit the number of reduce jobs in my application.

In the cluster, the maximum number of reduce jobs is 120. But, I don't want to use all of them, because my application doesn't need that many number of reduce jobs.

I tried the solution below, but it didn't change anything . My application still uses 120 reduce jobs.

How can I set the number of reduce jobs?

https://stackoverflow.com/questions/33237361/unable-to-set-mapreduce-job-reduces-through-generic-option-parser

Thanks for your response.

Community
  • 1
  • 1
seha
  • 51
  • 1

2 Answers2

2

From job class,

job.setNumReduceTasks(100);

From shell command

hadoop jar yourJar.jar -D mapred.reduce.tasks=1 <input> <output>
Ramzy
  • 6,948
  • 6
  • 18
  • 30
1

Please check below values

With 1.x

mapred.reduce.tasks will take precedence unless you specify job.setNumReduceTasks(x);

With 2.x

mapreduce.job.reduces will take precedence unless you specify job.setNumReduceTasks(x);

You can implement suggestion by Ramzy with -D parameter. Check the hadoop version and respective property before you set the property.

Have a look at deprecated properties and new properties between 1.x and 2.x versions

Ravindra babu
  • 37,698
  • 11
  • 250
  • 211