Parameter numPartitions in Partitioner class

Question

Gurus!

Can anybody answer: where defined parameter numPartitions in Partitioner class(value of this paramen)?

score 3 · Accepted Answer · answered Jun 18 '12 at 11:38

3

It's the number of reducers defined for your Map Reduce job

answered Jun 18 '12 at 11:38

Chris White

29,949
4
71
93

o, i thik so. But where i defined number of reducers in this example:http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html ? – Mijatovic Jun 18 '12 at 13:54
http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html#Job+Configuration - setNumReduceTasks(int) – Chris White Jun 18 '12 at 14:01
Oh, thanks. But i don`t understand - how many reduces i will have, if i will not use this parameter in the my code? In the mapred-site.xml i found parameter mapred.tasktracker.map.tasks.maximum. But i didn`t found the same parameter for reduce ... – Mijatovic Jun 18 '12 at 14:15
if you don't configure the parameter, and it's not configured in mapred-site.xml (`mapred.reduce.tasks`) then the default is 1. – Chris White Jun 18 '12 at 14:18
I found: http://hadoop.apache.org/common/docs/r0.20.2/mapred-default.html. So, if i not specify additional property, i will have only one reduce on Job. Example: i have 100 servers(datanode + tasktracker), i run MR job in default conf i will have 100 mapers (1 per node) and only one reducer per cluster. Is it correct? I'm sorry for alot amount foolish questions, but hadoop - revolution in the thinking – Mijatovic Jun 18 '12 at 14:31
yes, ifi you don't specify the property, you'll get one reducer for the job. The number of mappers is dependant on a number of things (input files, block sizes, 'splittability', etc), not the number of nodes in your cluster – Chris White Jun 18 '12 at 15:13

Shegit Brahm · Answer 2 · 2012-06-18T11:48:36.187

0

Do you mean that? numPartition in partition class. I just used search there.

* @param numPartitions the total number of partitions. (as integer)
* [...] The total number of partitions 
* is the same as the number of reduce tasks for the job.

edited Jun 18 '12 at 11:48

answered Jun 18 '12 at 11:05

Shegit Brahm

725
2
9
22

score 0 · Answer 3 · edited Aug 10 '14 at 15:23

0

In the Driver class you will mention the no of reducers required for the job. That count is called numPartion count. Like:

job.setNumReduceTasks(4);

edited Aug 10 '14 at 15:23

xav

5,452
7
48
57

answered Aug 10 '14 at 14:57

Arun Poreddy

11
2

Parameter numPartitions in Partitioner class

3 Answers3