Stack Overflow
Stack Overflow

Questions tagged [airflow-scheduler]

The Apache Airflow scheduler monitors all tasks and all DAGs, and triggers the task instances whose dependencies have been met, and Apache Airflow is a platform to programmatically author, schedule and monitor workflows.

1257 questions
12
votes
2 answers

Airflow 1.9.0 - Long Delay Between Task Execution

I recently upgraded from v1.7.1.2 to v1.9.0 and after the upgrade I noticed that the CPU usage increased significantly. After doing some digging, I tracked it down to these two scheduler config options: min_file_process_interval (defaults to 0) and…
flutikoff
  • 121
  • 1
  • 3
12
votes
2 answers

Airflow encrypted variables

After updating Airflow to 1.9 all variables are created as encrypted. Is it possible to disable encryption ?
Alexander Ershov
  • 1,105
  • 2
  • 12
  • 26
12
votes
4 answers

Airflow Worker Configuration

I am a newbie to Airflow. I'm trying to setup Distributed Mode of Airflow Using Celery Executor by Refering this article https://stlong0521.github.io/20161023%20-%20Airflow.html Before getting into detail about the specification I would like to…
Soundar Raj
  • 133
  • 1
  • 1
  • 6
12
votes
2 answers

Integration of Kubernetes with Apache Airflow

We are building workflow scheduling application. We found Airflow as a good option for workflow manager and Kubernetes as good option for Cluster manager. Thus, flow would be, We will submit workflow DAG to Airflow. Airflow should submit the tasks…
Free Coder
  • 459
  • 4
  • 19
12
votes
3 answers

How to trigger daily DAG run at midnight local time instead of midnight UTC time

I'm in the UTC+4 timezone, so when Airflow triggers the nightly ETLs, it's already 4:00AM here. How can I tell Airflow to trigger the run for day ds already on day ds-1 at 20:00, but with ds=ds? Per the docs it's highly recommended to keep all…
Marton Trencseni
  • 887
  • 2
  • 10
  • 18
11
votes
1 answer

Dynamic tasks in airflow based on an external file

I am reading list of elements from an external file and looping over elements to create a series of tasks. For example, if there are 2 elements in the file - [A, B]. There will be 2 series of tasks: A1 -> A2 .. B1 -> B2 ... This reading elements…
Dev
  • 13,492
  • 19
  • 81
  • 174
11
votes
2 answers

Event based Triggering and running an airflow task on dropping a file into S3 bucket

Is it possible to run an airflow task only when a specific event occurs like an event of dropping a file into a specific S3 bucket. Something similar to AWS Lambda events There is S3KeySensor but I don't know if it does what I want (to run Task only…
Kingz
  • 1,657
  • 3
  • 18
  • 29
11
votes
3 answers

Airflow: ValueError: Unable to configure handler 'processor' - wasb logger

I am trying to configure remote logging with Azure blob. Airflow version: 1.10.2 Python: 3.6.5 Ubuntu: 18.04 Following are the step I did: In $AIRFLOW_HOME/config/log_config.py, I have put REMOTE_BASE_LOG_FOLDER = 'wasb-airflow-logs' (This is a…
Gagan
  • 1,775
  • 5
  • 31
  • 59
11
votes
1 answer

DAGs not clickable on Google Cloud Composer webserver, but working fine on a local Airflow

I'm using Google Cloud Composer (managed Airflow on Google Cloud Platform) with image version composer-0.5.3-airflow-1.9.0 and Python 2.7, and I'm facing a weird issue : after importing my DAGs, they are not clickable from the Web UI (and there are…
norbjd
  • 10,166
  • 4
  • 45
  • 80
11
votes
2 answers

How to restart DAG in Airflow?

One of my DAGs with many steps stopped part way through because of a DB shutdown. I'd like to get the DAG to start where it left off but all I can do is start the individual tasks of the DAG one by one. Is there a way to tell Airflow to start the…
Paymahn Moghadasian
  • 9,301
  • 13
  • 56
  • 94
11
votes
3 answers

How to set a SLA in Airflow?

I would like to set an SLA in a Sensor Operator. The documentation is not too clear about the use of it. So I did a test using the S3KeySensor operator which is looking for a file that does not exist. I set the SLA to 30 seconds, I was hoping to see…
ultraInstinct
  • 4,063
  • 10
  • 36
  • 53
10
votes
5 answers

How to fix "Failed to fetch log file from worker. Unsupported URL protocol" error in Airflow DAG logs?

I am running Airflow via docker through this image apache/airflow:2.1.0 Please refer to this thread for the initial error I faced. Currently I am able to run my previous existing DAGs. However when I add newer DAGS I get the following error in the…
raaj
  • 403
  • 1
  • 5
  • 17
10
votes
2 answers

Airflow - Use TaskGroup and PythonBranchOperator in the same DAG

I am currently using Airflow Taskflow API 2.0. I am having an issue of combining the use of TaskGroup and BranchPythonOperator. Below is my code: import airflow from airflow.models import DAG from airflow.decorators import task, dag from…
adan11
  • 647
  • 1
  • 7
  • 24
10
votes
2 answers

Can an Airflow task dynamically generate a DAG at runtime?

I have an upload folder that gets irregular uploads. For each uploaded file, I want to spawn a DAG that is specific to that file. My first thought was to do this with a FileSensor that monitors the upload folder and, conditional on presence of new…
Simon Podhajsky
  • 307
  • 1
  • 3
  • 13
10
votes
7 answers

Airflow sla_miss_callback function not triggering

I have been trying to get a slack message callback to trigger on SLA misses. I've noticed that: SLA misses get registered successfully in the Airflow web UI at slamiss/list/ on_failure_callback works successfully However, the sla_miss_callback…
Allen Lee
  • 101
  • 1
  • 1
  • 3
1 2 3
…
83 84