The Apache Airflow scheduler monitors all tasks and all DAGs, and triggers the task instances whose dependencies have been met, and Apache Airflow is a platform to programmatically author, schedule and monitor workflows.
Questions tagged [airflow-scheduler]
1257 questions
-1
votes
1 answer
How to print Airflow time?
Need this info in the log as a print statement click for more info
Roshan_mobo
- 3
- 2
-1
votes
1 answer
How to call an external server from Apache Airflow ( just after every DAG run )
I am new to airflow, I want to call to an external licensing server every time a DAG run is successfully completed.
vajr1
- 27
- 5
-1
votes
1 answer
Start Airflow Webserver Automaticalll as a Service
I currently would like to write a system configuration file that starts the airflow webserver. Here is my config file definition:
#airflow-webserver.service
[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service…
Lazloo Xp
- 858
- 1
- 11
- 36
-1
votes
1 answer
Airflow Task unexpected keyword Issue
I have a python operator.
PythonOperator(
task_id="getData",
python_callable=getData,
op_kwargs={
"params": {
"days": 1,
"limit": 10
}
…
Avenger
- 793
- 11
- 31
-1
votes
1 answer
How make branches in a Apache Airflow Dag?
I have a dag like this (This is a semi-pseudocode), I want to execute the tasks in different branches based on their output.
#This is a method that return a or b
def dosth():
.....
return a or b
t1 = PythonOperator(
't1',
…
yuchennnnn
- 21
- 1
- 2
-1
votes
1 answer
how to stop my dependent tasks if earlier task fails
Hi I want to set up Airflow job schedule... where I have 3 tasks i.e. task_1 > tast_2 > task_3.
If first task , task_1 fails I need to stop remaining tasks being executed.
How this can be handled ?
How can I know earlier job failed or succeeded ?…
BdEngineer
- 2,929
- 4
- 49
- 85
-1
votes
1 answer
how to get airflow schedule right
I found very confusing about how the airflow schedule works.
I would like to schedule a dag that runs on Friday and I would like use its result on Saturday. So I did the crontab expression like this: 00 16 * * 5, however, as of today 2020-03-10, the…
JOHN
- 871
- 1
- 12
- 24
-1
votes
2 answers
airflow + why airflow return unclear erros
I am using the airflow application on my Linux server
note - Airflow is a platform to programmatically author, schedule and monitor workflows.
from some unclear reason we get the following issues ( when we restart the airflow or initialize the…
Judy
- 1,595
- 6
- 19
- 41
-1
votes
3 answers
How to run a Talend .jar job on Airflow?
Hello people of the Earth! I'm using Airflow to schedule and run .jar Job generated with Talend Open Studio BigData,i added my job as a DAG in Airflow by making a .py file under AIRFLOW_HOME/dags :
from airflow import DAG
from…
Yassin Abid
- 315
- 2
- 5
- 12
-2
votes
1 answer
xcom_pull is not working in dynamic task generation function
I have a function which dynamically creates a sub task, where I am reading value from xcom_pull but there I am getting the error:
File "/home/airflow/gcs/recon_nik_v6.py", line 168, in create_audit_task
…
Yug
- 105
- 1
- 9
-2
votes
1 answer
Can Spark jobs be scheduled through Airflow
I am new to spark and need to clarify some doubts i have.
Can I schedule Spark Jobs through Airflow
My Airflow (Spark) jobs process raw csv files present in S3 bucket and then transforms into parquet format , stores it into S3 bucket and then…
Siddharth Kumar
- 3
- 1
-2
votes
1 answer
How to submit spark-submit from Apache Airflow
Can anyone help me how to schedule a spark job in the Apache Airflow,
I am looking for the script please help me
