Hive unable to manually set number of reducers

Question

I have the following hive query:

select count(distinct id) as total from mytable;

which automatically spawns:
1408 Mappers
1 Reducer

I need to manually set the number of reducers and I have tried the following:

set mapred.reduce.tasks=50 
set hive.exec.reducers.max=50

but none of these settings seem to be honored. The query takes forever to run. Is there a way to manually set the reducers or maybe rewrite the query so it can result in more reducers? Thanks!

It doesn't matter Tudor, even if he had only reduce slot, he could still have more reducers. — Donald Miner, Jan 06 '12 at 17:51
I doubt this is true since you have 1400 mappers, but are you running in local mode? If so, that'll keep your reducer at 1, I believe. — Donald Miner, Jan 06 '12 at 18:04

score 46 · Accepted Answer · answered Jan 07 '12 at 14:58

46

writing query in hive like this:

 SELECT COUNT(DISTINCT id) ....

will always result in using only one reducer. You should:

use this command to set desired number of reducers:

set mapred.reduce.tasks=50
rewrite query as following:

SELECT COUNT(*) FROM ( SELECT DISTINCT id FROM ... ) t;

This will result in 2 map+reduce jobs instead of one, but performance gain will be substantial.

answered Jan 07 '12 at 14:58

wlk

5,695
6
54
72

cool. how comes the hive compiler doesn't do this optimization (turning into 2 MR jobs) by itself automatically? – ihadanny Apr 26 '13 at 21:22
There are situations where turning this into 2 MR jobs isn't an optimization. For instance, if id is already close to unique and the table is stored in a columnar file format (like RCFILE), than 1 MR job would certainly be better. Since situations like that aren't outlandish, I imagine that's why no one has built this optimization into Hive. – Daniel Koverman May 16 '13 at 19:59

user1314742 · Answer 2 · 2016-03-16T13:16:59.690

11

Number of reducers depends also on size of the input file

By default it is 1GB (1000000000 bytes). You could change that by setting the property hive.exec.reducers.bytes.per.reducer:

either by changing hive-site.xml

<property>
   <name>hive.exec.reducers.bytes.per.reducer</name>
   <value>1000000</value>
</property>

or using set

$ hive -e "set hive.exec.reducers.bytes.per.reducer=1000000"

edited Mar 16 '16 at 13:16

answered Mar 18 '14 at 18:23

user1314742

2,865
3
28
34

score 1 · Answer 3 · answered Mar 13 '18 at 05:16

Mapper is totaly depend on number of file i.e size of file we can call it as input splits. Split is noting but the logical split of data. Ex: my file size is 150MB and my HDFS default block is 128MB. It will create two split means two blocks. Two Mapper will get assigned for this job.

Imp Note: Suppose I have specified the split size is 50MB then It will start 3 Mapper because of it totally depend on number of split.

Imp Note: if you expect 10TB of input data and have a blocksize of 128MB, you’ll end up with 82,000 maps, unless Configuration.set(MRJobConfig.NUM_MAPS, int) (which only provides a hint to the framework) is used to set it even higher.

Note: If we haven't specifyed the split size it will take default hdfs block size as split size.

Reducer has 3 primary phases: shuffle, sort and reduce.

Command :

1] Set Map Task : -D mapred.map.tasks=4
2] Set Reduce Task : -D mapred.reduce.tasks=2

score 1 · Answer 4 · answered Jan 06 '12 at 18:10

1

You could set the number of reducers spawned per node in the conf/mapred-site.xml config file. See here: http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html.

In particular, you need to set this property:

mapred.tasktracker.reduce.tasks.maximum

answered Jan 06 '12 at 18:10

Tudor

61,523
12
102
142

that is applicable for all jobs. If you want to set for a specific query, i think it is better use `set mapred.reduce.tasks` – brain storm Aug 06 '14 at 17:32

Hive unable to manually set number of reducers

4 Answers4

Linked