Issue with HIVE query (Select count(*))

Question

I created a hive table as in Create external with Partition

I am using hive version of hive-0.7.1-cdh3u2. When i ran simple query i.e. select count(*) i am getting error.

hive> select count(*) from test where dt='2012-01-30' and hr='17';
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201201311809_0006, Tracking URL = http://localhost:15030/jobdetails.jsp?jobid=job_201201311809_0006
Kill Command = /Users/balaji/svn/app/hadoop/hadoop-0.20.2-cdh3u2/bin/hadoop job  -Dmapred.job.tracker=localhost:10012 -kill job_201201311809_0006
2012-01-31 21:20:58,074 Stage-1 map = 0%,  reduce = 0%
2012-01-31 21:21:25,402 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201201311809_0006 with errors

And the jobtracker error is

FAILED  

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.apache.hadoop.hive.shims.Hadoop20SShims$CombineFileRecordReader.initNextRecordReader(Hadoop20SShims.java:306)
    at org.apache.hadoop.hive.shims.Hadoop20SShims$CombineFileRecordReader.<init>(Hadoop20SShims.java:269)
    at org.apache.hadoop.hive.shims.Hadoop20SShims$CombineFileInputFormatShim.getRecordReader(Hadoop20SShims.java:366)
    at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:413)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:371)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
    at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
    at sun

Can someone please help. I am totally blocked because of this issue. Thanks!

does it also happen with non-external tables? Which version of hadoop are you using? — wlk, Feb 01 '12 at 11:44
I have not tried with non-external tables. I am using hadoop-0.20.2-cdh3u2. — Balaji, Feb 01 '12 at 17:33
Hi Wojtek - The issue occurs even in non-external tables. I assume the issue is w.r.t dependencies but not sure which one causes it. — Balaji, Feb 01 '12 at 17:59
@Wojtek - I am not using LZO compression. When i used the cloud era hive, the mr started working. Thanks for your help! — Balaji, Feb 02 '12 at 07:29

score 3 · Accepted Answer · answered Feb 08 '12 at 07:26

3

I found the issue. The hive was looking for Serde jar to execute it in Hadoop. Got fixed by adding the property in hive-default.xml

<property>
   <name>hive.aux.jars.path</name>
   <value>serde jar path in hadoop</value>
</property>

Thank you

answered Feb 08 '12 at 07:26

Balaji

859
1
16
27

Issue with HIVE query (Select count(*))

1 Answers1