Stack Overflow
Stack Overflow

Questions tagged [oozie]

Oozie is a workflow/coordination system to manage Hadoop Map Reduce jobs

Oozie is a workflow scheduler system to manage Apache Hadoop jobs.

Oozie Workflow jobs are Directed Acyclical Graphs (DAGs) of actions.

Oozie Coordinator jobs are recurrent Oozie Workflow jobs triggered by time (frequency) and data availabilty.

Oozie is integrated with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (such as Java map-reduce, Streaming map-reduce, Pig, Hive, Sqoop and Distcp) as well as system specific jobs (such as Java programs and shell scripts).

Oozie is a scalable, reliable and extensible system.

References

Related Tags

1929 questions
0
votes
1 answer

Oozie Job getting Suspended and not reaching YARN

I am trying to start a Oozie Shell Action Job via cli as: oozie job -config jobprops/jos.prioperties -run The Job Starts, it gives me a unique Id and I can see Job in Oozie UI. However, Yarn Console shows no submitted jobs and on checking log in…
Vikas Raturi
  • 916
  • 2
  • 13
  • 23
0
votes
1 answer

could not resolve org.apache.hcatalog.pig.hcatloader using imports

I am trying to create a workflow with just one pig action, I have all the jars available in the path /user/oozie/share/lib/lib_* and also I am defining the path of hive-site.xml in the job nodes. But the job fails saying : could not resolve…
Immanuel Fredrick
  • 508
  • 3
  • 9
  • 20
0
votes
1 answer

Using Environment variables with cloudera manager

What would be the best way to use an environment variables declared for different users in a cluster(all nodes) and make a call to a oozie workflow (Cloudera) and the container of yarn recover the environment variable according to the user. In the…
David Gomez
  • 33
  • 4
0
votes
0 answers

Oozie Error Code: E1100 & ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

Hello I am trying to do this example Hadoop Data Pipeline ...here I am running a Flume agent where Flume copies files from local to HDFS & Falcon does the job of processing data files & after data is processed,Hive processing lineage will be…
Akki
  • 493
  • 1
  • 11
  • 23
0
votes
0 answers

In pig script, how to save passed in parameters/arguments

In pig script, I know that we can use passed-in parameters like the following: %declare CMD `$mycmd $date` A = load '/data/mydata/$CMD'; B = filter A by $0>'5'; But how can I save the parameters into a file? Thanks
RhysJ
  • 153
  • 1
  • 3
  • 19
0
votes
0 answers

Oozie job running hive action throwing Error on pathname

Here's the error I'm receiving 13288 [main] ERROR hive.ql.metadata.Hive - MetaException(message:java.lang.IllegalArgumentException: Pathname /apps/hive/warehouse/my_db.db/clog/${clogDataOutputDir}/logmessages.log.${hiveconf:current_date} from …
tchoedak
  • 87
  • 1
  • 2
  • 10
0
votes
2 answers

Sqoop action using Oozie in Hue

I am using AWS EMR and I am unable to submit Sqoop action through Oozie in Hue.I have installed sqoop and can run a sqoop job from cli and not through Oozie action in Hue. The error that I get is "sqoop command not found" Can anyone help me out with…
Bharath
  • 467
  • 2
  • 8
  • 20
0
votes
0 answers

Retrieving action output in oozie with wf:actionData property

I'm writing out some messages to oozie.action.output.properties file in java action called through oozie workflow. That message is retrieved in next workflow using ${wf:actionData('')['']}. But, sometimes I need not write any…
Mike
  • 725
  • 2
  • 11
  • 21
0
votes
1 answer

Oozie hourly coordinator timing out on future actions

At the 5 minute mark of every hour, I have data from the past hour loaded into hdfs. I thought I could setup a coordinator job to run at 10 minute mark of every hour to process this data while doing a check if the directory for that hour exists.…
tchoedak
  • 87
  • 1
  • 2
  • 10
0
votes
2 answers

Oozie Setup does not work

Yet another problem with Apache Bigtop. This time it is oozie. Whenever I want to do something with Oozie, I get the following exception: java.lang.IllegalArgumentException: Oozie URL is not available neither in command option or in the…
Nelnel
  • 61
  • 1
  • 5
0
votes
1 answer

Job started with MapReduce gets killed .Why?

i try for several days to start a wordount(MapReduce) job with Oozie. With normal(CMD: "hadoop jar *.jar mainClass input output") job start all things goes just fine . Current oozie configuration is…
Mihai Morareanu
  • 1
  • 2
0
votes
1 answer

Hue: oozie parameters

I want to pass 2 parameters to my Hiveql script in oozie, my script: ALTER TABLE default.otarie_appsession ADD IF NOT EXISTS PARTITION ( insert_date=${dt},hr=${hr} ); My Oozie workflow : When i send the job it ask for parameter values, so i…
54l3d
  • 3,913
  • 4
  • 32
  • 58
0
votes
2 answers

Oozie job not working

I've been trying to run an Oozie job since last few days but its not running. The gets submitted but remain stuck in the "running" state in the Oozie Web console. I'm using using Oozie 4.1.0 and Hadoop 2.6 for this configuration. I checked Oozie…
Syed Rizvi
  • 41
  • 6
0
votes
1 answer

Oozie Sqoop job - cannot restore job

On HDP 2.3.4, using Oozie 4.2.0 and Sqoop 1.4.2, I'm trying to create a coordinator app that will execute sqoop jobs on a daily basis. I need the sqoop action to execute jobs because these are incremental imports. I've configured sqoop-site.xml and…
Ton Torres
  • 1,509
  • 13
  • 24
0
votes
1 answer

Why Oozie allocates more memory when running MapReduce jobs?

I'm running MapReduce jobs using oozie. From workflow i'm just invoking MapReduce driver class and nothing other than that. But for this oozie workflow takes lot of memory. It needs minimum of 2GB container size to invoke the driver class. Below is…
Vijayakumar
  • 303
  • 4
  • 10
1 2 3
…
99
100