Stack Overflow
Stack Overflow

Questions tagged [airflow-scheduler]

The Apache Airflow scheduler monitors all tasks and all DAGs, and triggers the task instances whose dependencies have been met, and Apache Airflow is a platform to programmatically author, schedule and monitor workflows.

1257 questions
19
votes
3 answers

how do I use the --conf option in airflow

I am trying to run a airflow DAG and need to pass some parameters for the tasks. How do I read the JSON string passed as the --conf parameter in the command line trigger_dag command, in the python DAG file. ex: airflow trigger_dag 'dag_name' -r…
Praveen Singh
  • 219
  • 1
  • 2
  • 7
19
votes
2 answers

Airflow: How to push xcom value from PostgreOperator?

I'm using Airflow 1.8.1 and I want to push the result of a sql request from PostgreOperator. Here's my tasks: check_task = PostgresOperator( task_id='check_task', postgres_conn_id='conx', sql="check_task.sql", xcom_push=True, …
Omar14
  • 2,007
  • 4
  • 21
  • 34
19
votes
5 answers

Airflow: New DAG is not found by webserver

In Airflow, how should I handle the error "This DAG isn't available in the webserver DagBag object. It shows up in this list because the scheduler marked it as active in the metadata database"? I've copied a new DAG to an Airflow server, and have…
Ollie Glass
  • 19,455
  • 21
  • 76
  • 107
18
votes
2 answers

airflow pass parameter from cli

Is there a way to pass a parameter to: airflow trigger_dag dag_name {param} ? I have a script that monitors a directory for files - when a file gets moves into the target directory I want to trigger the dag passing as a parameter the file path.
bsd
  • 1,207
  • 4
  • 15
  • 28
17
votes
5 answers

Airflow: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set

I've setup airflow 2.0 on my local mac, backed with PostgresSQL for sql_alchemy_conn. My DAG's are running fine + maintaining metadata in sqla database, but any airflow users command errors with roles table/views not created. I've tried setting and…
appleboy
  • 661
  • 1
  • 9
  • 15
17
votes
9 answers

Airflow unpause dag programmatically?

I have a dag that we'll deploy to multiple different airflow instances and in our airflow.cfg we have dags_are_paused_at_creation = True but for this specific dag we want it to be turned on without having to do so manually by clicking on the UI. Is…
Ace Haidrey
  • 1,198
  • 2
  • 14
  • 27
16
votes
2 answers

dag.py raises: "airflow.exceptions.AirflowException: Task is missing the start_date parameter", but its given in code

i today tried to create my first airflow DAG: from datetime import timedelta from airflow import DAG from airflow.operators.bash_operator import BashOperator from airflow.operators.dummy_operator import DummyOperator from…
Luigi Drago
  • 163
  • 1
  • 1
  • 5
16
votes
2 answers

Airflow latency between tasks

As you can see in the image : airflow is making too much time between tasks execution ? it almost represents 30% of the DAG execution time. I've changed the airflow.cfg file to: job_heartbeat_sec = 1 scheduler_heartbeat_sec = 1 but I still have…
I.Chorfi
  • 507
  • 2
  • 5
  • 12
15
votes
4 answers

How to trigger airflow dag manually?

I have beening working on Airflow for a while for no problem withe the scheduler but now I have encountered a problem. Bascially I have a script and dag ready for a task, but the task doesn't run periodically. Instead it needs to be activated at…
user1799438
  • 151
  • 1
  • 1
  • 3
15
votes
5 answers

How to stop/kill airflow scheduler started in daemon mode

I and new to airflow and accidentally started airflow scheduler in daemon mode. Now, I want to kill the scheduler and possibly restart it. I tried doing sudo kill -9 pkill nothing is happening. When I run ps aux |…
yguw
  • 856
  • 6
  • 12
  • 32
14
votes
2 answers

Airflow: Trigger DAG via UI with Parameters/Config

I see that one can trigger_dag with parameters/config key-value pairs using the airflow command line: For Apache Airflow, How can I pass the parameters when manually trigger DAG via CLI? In my case, I would like to centralize all operations for…
chaooder
  • 1,358
  • 1
  • 17
  • 37
14
votes
8 answers

Errno 13 Permission denied when Airflow tries to write to logs

We're running into a permission error when using Airflow, receiving the following error: PermissionError: [Errno 13] Permission denied: '/usr/local/airflow/logs/scheduler/2019-12-18/../../../../home We've tried using chmod 777 -R on the…
phenderbender
  • 625
  • 2
  • 8
  • 18
14
votes
3 answers

Airflow scheduler stuck

I'm testing the use of Airflow, and after triggering a (seemingly) large number of DAGs at the same time, it seems to just fail to schedule anything and starts killing processes. These are the logs the scheduler prints: [2019-08-29 11:17:13,542]…
GuD
  • 668
  • 5
  • 9
14
votes
1 answer

How to resolve DB connection invalidated warning in Airflow Scheduler?

I am upgrading our Airflow instance from 1.9 to 1.10.3 and whenever the scheduler runs now I get a warning that the database connection has been invalidated and it's trying to reconnect. A bunch of these errors show up in a row. The console also…
trker
  • 412
  • 3
  • 14
14
votes
2 answers

Re-run part of an Airflow Subdag

I have a daily Dag that contains a subdag. The subdag has five tasks, T1 through T5, that must run in order (e.g. T1 >> T2 >> T3 >> T4 >> T5) The dag successfully runs for a few days but then I discover a bug with T4. I fix the bug and want to…
stipe108
  • 1,640
  • 1
  • 16
  • 20
1
2
3
…
83 84