Azkaban - HIve Job Failed

Question

I am trying to run hive job in azkaban

I was able to upload the hive job but the hive job is not getting executed properly.

ERROR [hive-demo] [Azkaban] Failed to build job executor for job hive-demoJob type     'hive' is unrecognized. Could not construct job[{hive.query.01: drop table words;, hive.query.03: describe words;, hive.query.02: create table words (freq int, word string) row format delimited fields terminated by ' ' stored as textfile;, working.dir: /home/hduser/Technology/azkaban/azkaban-executor/executions/44, azk.hive.action: execute.query, azkaban.job.attempt: 0, type: hive, hive.query.05: select * from words limit   10;, hive.query.04: load data local inpath "res/input" into table words;, user.to.proxy:   azkaban, hive.query.06: select freq, count(1) as f2 from words group by freq sort by f2 desc  limit 10;, parent = {azkaban.flow.flowid: hive-demo, azkaban.flow.execid: 44,  azkaban.flow.start.timezone: America/Los_Angeles, azkaban.flow.start.hour: 22,  azkaban.flow.start.second: 17, azkaban.flow.start.year: 2014,  azkaban.flow.start.milliseconds: 111, azkaban.flow.start.minute: 54,  azkaban.flow.start.timestamp: 2014-01-31T22:54:17.111-08:00, azkaban.flow.start.month: 01,  azkaban.flow.projectversion: 2, azkaban.flow.projectid: 4, azkaban.flow.uuid: 29a46293-cc95- 4c66-b2b1-51014205ce3b, azkaban.flow.start.day: 31, parent = {}}}] of type[hive]. 2014/02/01  12:24:17.295 +0530 ERROR [hive-demo] [Azkaban] Failed to build job type, skipping this job    2014/02/01 12:24:17.295 +0530 ERROR [hive-demo] [Azkaban] Job run failed! 2014/02/01 12:24:17.295 +0530 INFO [hive-demo] [Azkaban] Finishing job hive-demo at 1391237657295  2014/02/01 12:24:17.382 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning recently finished  2014/02/01 12:24:17.383 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning execution 43 from  recently finished flows list. 2014/02/01 12:24:17.408 +0530 INFO [hive-demo] [Azkaban] Job  Finished hive-demo with status FAILED 2014/02/01 12:24:17.463 +0530 INFO [hive-demo]  [Azkaban] Finishing up flow. Awaiting Termination 2014/02/01 12:24:17.463 +0530 INFO [hive- demo] [Azkaban] Setting flow status to Failed. 2014/02/01 12:24:17.464 +0530 INFO [hive- demo] [Azkaban] Flow is set to FAILED 2014/02/01 12:24:17.464 +0530 INFO [hive-demo]  [Azkaban] Setting end time for flow 44 to 1391237657464 2014/02/01 12:24:17.552 +0530 INFO  [FlowRunnerManager] [Azkaban] Flow 44 is finished. Adding it to recently finished flows  list. 2014/02/01 12:26:17.383 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning recently  finished 2014/02/01 12:26:17.384 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning execution  44 from recently finished flows list.
 ========================================================================

Here is the content in my plugin.properties

job.class=azkaban.jobtype.hiveutils.azkaban.HiveViaAzkaban

hive.home=HIVE_HOME

hive.aux.jars.path=/usr/lib/hive/lib
env.HIVE_HOME=$HIVE_HOME
env.HIVE_AUX_JARS_PATH=/usr/lib/hive/lib

#hive.jvm.args=-Dhive.querylog.location=. -Dhive.exec.scratchdir=/tmp/hive-    ${user.to.proxy} -Dhive.aux.jars.path=${hive.aux.jars.path}
jobtype.global.jvm.args=${hive.jvm.args}

this is my private.properties jobtype.classpath=$HADOOP_HOME/conf,$HADOOP_HOME/lib/*,$HIVE_HOME/lib/*,$HIVE_HOME/conf,/usr /lib/hive/lib jobtype.class=azkaban.jobtype.HadoopJavaJob

hive.aux.jars.path=/usr/lib/hive/lib
env.HADOOP_HOME=$HADOOP_HOME
env.HIVE_HOME=$HIVE_HOME
env.HIVE_AUX_JARS_PATH=/usr/lib/hive/lib

hive.home=$HIVE_HOME

jobtype.global.classpath=${hive.classpath.items}
  hive.classpath.items=./*,${hive.home}/lib/*,${hive.home}/conf/,${hive.aux.jars.path}/*,${had    oop.home}/*,${hadoop.home}/lib/*,${hadoop.home}/conf/

I have added the plugin folder in executor folder only.

Why is not detecting the job type. please help.

Answer 1

It's hard to tell what is going on since you neither share the job itself nor logs, but try the following and let us know if this works. Create a file test.job with the following content:

type=hive
azk.hive.action=execute.query
hive.query=CREATE TABLE test(id INT, name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
user.to.proxy=hadoop

and submit it as described in the 'creating flow' section. For further details on the different types of jobs Azkaban (2.1) supports out of the box see the job types/plugin documentation.