Stack Overflow
Stack Overflow

Questions tagged [azkaban]

Azkaban is a batch workflow job scheduler created at LinkedIn to run their Hadoop Jobs.

Often times there is a need to run a set of jobs and processes in a particular order within a workflow. Azkaban will resolve the ordering through job dependencies and provide an easy to use web user interface to maintain and track your workflows. Here are a few features:

  • Compatible with any version of Hadoop
  • Easy to use web UI
  • Simple web and http workflow uploads
  • Project workspaces
  • Scheduling of workflows
  • Modular and pluginable
  • Authentication and Authorisation
  • Tracking of user actions
  • Email alerts on failure and successes
  • SLA alerting and auto killing
  • Retrying of failed jobs

http://azkaban.github.io

64 questions
1
vote
1 answer

How to schedule commands to run in background in Azkaban

In bash script, I can append & at the end of the command to make it run in the background such as: top & However, when I put that in a Azkaban job, such as: type=command command=top & It will give the error message as: 29-04-2015 23:53:08 PDT…
Gongqin Shen
  • 753
  • 6
  • 13
1
vote
0 answers

PHP script stopped unexpectedly without error

This script is executed by job scheduler Azkaban every day and have been working fine for more than a week now. But yesterday it failed unexpectedly. No exception was thrown, no fatal error and no error log entry. I am inserting a count of records…
Jya
  • 763
  • 6
  • 22
1
vote
1 answer

Azkaban - HIve Job Failed

I am trying to run hive job in azkaban I was able to upload the hive job but the hive job is not getting executed properly. ERROR [hive-demo] [Azkaban] Failed to build job executor for job hive-demoJob type 'hive' is unrecognized. Could not…
user2728024
  • 1,496
  • 8
  • 23
  • 39
0
votes
0 answers

Is it possible to configure dependencies in Azkaban to start a job after completion of either Job A or Job B, without requiring both of them to finish

I have a scenario where I have three jobs in my Azkaban workflow. I want to ensure that Job C starts only after the completion of either Job A or Job B. It doesn't matter which of the two jobs finishes first; as soon as either Job A or Job B…
Bora Çolakoğlu
  • 69
  • 2
  • 7
0
votes
0 answers

Include Flow parameters in the failure email whenever a flow fails

I have a use case where I am using Azkaban Ajax calls to trigger a flow by providing flowOverride parameters. There will be multiple concurrent executions of this flow with around 100 different flow parameters. Every time one of the execution fails,…
Ishan Anirudh Joshi
  • 23
  • 6
0
votes
0 answers

How to globally set a command to execute before each job in Azkaban without modifying project zip files?

I'm working with Azkaban and I'd like to globally set a command to be executed before each job with the type "command". Specifically, I want to insert a piece of code at the beginning of the bash script that is to be executed in the command. I've…
Jacob Cao
  • 1
0
votes
0 answers

Azkaban 3.44 conditional flow not working (including example on official documentation)

I'm try to use conditional flows on Azkaban. When I submit/upload my project inside web node I receive this error. Validator Directory Flow reports errors: Error loading flow yaml file sample.flow:Cannot create property=nodes for…
Holsi
  • 21
  • 3
0
votes
0 answers

Azkaban 3.41 generating same execution id for different flow runs

When I run azkaban flow, sometimes it assigns same execution id which is assigned to same other flow execution. Ideally each flow execution should be assigned with unique execution id, but this is not happening in my case. Example Lets say we have…
Aditya Calangutkar
  • 486
  • 1
  • 6
  • 21
0
votes
0 answers

Github Actions fails on deploying to Azkaban

I'm deploying app with github actions and getting error when trying to deploy to Azkaban with docker image. Any ideas what I should do? ./docker-run.sh: 4: cd: can't cd to target/deployment chmod: cannot access './script.sh': No such file or…
Anja
  • 1
  • 1
0
votes
0 answers

Apache Airflow/Azkaban workflow Schedulers compatibility with Hadoop MRv1

I'm working on a project that relies on Hadoop but MRv1 architecture (Hadoop-1.1.2). I tried oozie scheduler for creating workflows(mapred) but gave up eventually, cause it is a nightmare to configure and I couldn't get it to work. I was wondering…
aniii
  • 3
  • 2
0
votes
1 answer

azkaban run selenium automatic python script flow failed when after execute about twenty miniutes,and the system becomes very slowly

I run python script in azkaban. enviroment: CentOS 8.1 azkaban 3.90.0 Python 3.6.8 ChromeDriver84.0.4147.30 In test.flow file nodes: - name: job_test type: command config: command: python3…
Venus
  • 1,184
  • 2
  • 13
  • 32
0
votes
1 answer

azkaban install gradlew specify java location

environment:CentOS 8.1 64bit,azkaban 3.90.0 ,java 14.0.1 sudo adduser azkaban cd /home/azkaban git clone git@github.com:azkaban/azkaban.git git checkout tags/3.90.0 -b 3.90.0-branch when I run cd /home/azkaban/azkaban ./gradlew clean build it…
fabby
  • 11
  • 2
0
votes
0 answers

HADOOP_HOME not set while running Azkaban Solo Server?

I am trying to run Azkaban Solo Server. Using the steps:Azkaban Solo Get Started I am running the start-solo.sh, but can't get anything on localhost:8081. I am trying to figure the issue using the log generated at…
Prakhar Gurawa
  • 373
  • 3
  • 13
0
votes
1 answer

azkaban h2.mv.db is too big and is that a problem?

we have used azkaban about 4 month and found that the h2.mv.db file is so big. Is this a problem need to be solved? How can we reduce the size of h2? and is there any way to transform into mysql? many thanks!
james.peng
  • 373
  • 1
  • 3
  • 13
0
votes
1 answer

azkaban build failed: Could not resolve all files for configuration ':azkaban-common:compileClasspath'

Ubuntu 14.04.5 LTS when run ./gradlew clean build ,it returns : FAILURE: Build failed with an exception. * What went wrong: Could not resolve all files for configuration ':azkaban-common:compileClasspath' …
Venus
  • 1,184
  • 2
  • 13
  • 32
1 2
3
4 5