0

Recently we have upgraded from Hadoop 2.0.0-cdh4.2.1 to Hadoop 2.6.0-cdh5.4.2. Now we are using Hive 1.1.0-cdh5.4.2.

When I ran a simple hive query it's taking too many reducers, In the previous version it took 120 reducers and in the new version it took 1100 reducers.

Can any one tell me why this is happening ?

Thanks in advance.

user3792699
  • 339
  • 3
  • 7
  • 17
  • Please add, the query you are executing. – Ramzy Jun 08 '15 at 13:07
  • Query: select id1, day, seq, count(1) from table_name where 1=1 and concat(day,hour)>='2015-05-3004' and concat(day,hour)<'2015-06-0604' and id2 in ('11111','2222','3333') group by id1, day, seq – user3792699 Jun 08 '15 at 13:12

1 Answers1

0

The number of reducers is decided by hive, based on bytes you allocate or the type of query you used (usage of count and just select *). Please look here for more information. here

Community
  • 1
  • 1
Ramzy
  • 6,948
  • 6
  • 18
  • 30
  • When I ran the same query before upgradation it took 120 reducers and after upgradation it is taking 1100 reducers. – user3792699 Jun 08 '15 at 13:25
  • Is the config parameter(ive.exec.reducers.bytes.per.reducer) same before and after upgradation? – Ramzy Jun 08 '15 at 13:36
  • [This](http://stackoverflow.com/questions/8762064/hive-unable-to-manually-set-number-of-reducers) post too can help you understand more. Happy Learning and coding – Ramzy Jun 08 '15 at 13:47