Stack Overflow
Stack Overflow

Questions tagged [airflow-2.x]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.

690 questions
0
votes
0 answers

Custom HTML template is not working on Airflow

Am trying to use a custom body template for the alert emails on airflow. Have created a file with the below message for testing purpose and mapped the file path to the variable html_content_template in the airflow.cfg file Try {{try_number}} out of…
Manigandan
  • 5,004
  • 1
  • 28
  • 47
0
votes
2 answers

Execute cleanup task on upstream task failure

I'm using the taskflow API on Airflow v2.3.0. I have a DAG that has a bunch of tasks. The first task (TaskA) turns on an EC2 instance and one of the last tasks (TaskX) turns off the EC2 instance. TaskA returns the instance ID of the EC2 instance,…
inspectorG4dget
  • 110,290
  • 27
  • 149
  • 241
0
votes
1 answer

How to get the broker tab in flower working with Airflow and RabbitMq over SSL?

In Flower the broker tab is not showing the queues when using it together with Airflow 2.3.4, RabbitMQ over SSL and Celery Executor As far as I understand Flower needs two parameters to make this tab work: broker_url, e.g.…
herrjeh42
  • 2,782
  • 4
  • 35
  • 47
0
votes
1 answer

Can TriggerDagRunOperator have a conf that is a number derived from a xcom or task?

I'm newer to airflow, but I'm having difficulties really understanding how to pass small xcom values around. Sometimes, this seems to work without an issue; other times, it takes me hours. Here's an example that has really been getting me. I'm…
radiomime
  • 118
  • 1
  • 1
  • 11
0
votes
1 answer

Does temporary files created exist across tasks in an airflow DAG

Does airflow DAG retains temp file created across tasks. My code is //This task calls an external api and downloads data into download.csv, this temp file download.csv is created on the fly by task using open function in python t1 = BashOperator( …
Deepak Kothari
  • 1,601
  • 24
  • 31
0
votes
0 answers

How to create Airflow role that can see or change Airflow DAG which owner is xxx

I am using Airflow 2.0.2. I have some DAGs that owner is "test". default_args = { "owner": "test", } How can I create Airflow role that can see or change Airflow DAGs that owner is "test"? This document explain DAG level access control but I do…
Yohei Onishi
  • 1,362
  • 1
  • 19
  • 42
0
votes
0 answers

Re-run airflow historical runs

I have a dag with the following parameters: start_date=datetime(2020, 7, 6) schedule_interval="0 12 * * *", concurrency=2, max_active_runs=6, catchup=True I had to re-process a year's historical data, so I did a reset of dag run…
Lee
  • 2,874
  • 3
  • 27
  • 51
0
votes
1 answer

Apache airflow running issue in browser (favicon.ico)

I am new to apache airflow. I installed airflow locally in my laptop using Windows subsystem for linux, after installation of Python 3 pip and other Dependancies I hit command "airflow webserver -p 8080" on ubuntu terminal. Airflow starts normally,…
rahul yadav
  • 11
  • 1
0
votes
0 answers

Airflow - How to properly define the time my DAG will execute every day?

I have two dags: The first one extracts data from one database to another. I want it to run everyday at 4 AM and then, that's how I defined my params: Note: The code has 7am instead of 4 because Airflow is in UTC time and my time is GMT-3. …
Luan Lopes
  • 1
0
votes
1 answer

Doesn't the airflow ecosystem share a single root logger?

I am trying to ship all airflow logs to kafka by attaching a new handler to the root logger, but not all logs are being published. Do I need to configure something else here? This is what I'm doing: custom_log_config.py LOGGING_CONFIG =…
Aman
  • 115
  • 1
  • 10
0
votes
1 answer

Get Airflow DAG Task Dependencies from Airflow Metadata

We are trying to analyze our DAG tasks over time, and want to be able to query the data in Airflow's (v2) metadata database. A simple example would be a DAG with a > b > c. By querying the TaskInstance metadata, and we can get all the individual…
Ben L
  • 83
  • 8
0
votes
1 answer

Something wrong with database initialized by Airflow

I'm trying to work with Apache Airflow. When I started my first DAG, it fails (the very first task fails). When I try to check logs, they are not displayed properly in the GUI and I get the exception 'airflow.models.taskinstance.TaskInstance object'…
Александр М
  • 306
  • 2
  • 10
0
votes
0 answers

Pause an Airflow DAG that runs every 10 minutes for 45 min

I have an Airflow DAG that runs for every 10 min. My cron tab expression looks like this */10 1-8,10-23 * * *. This currently pauses the DAG for 1 hours(9th hour UTC). I want to change this to pause the DAG for 45 minutes from 12:30 to 1:15 AM…
Vaishak
  • 607
  • 3
  • 8
  • 30
0
votes
1 answer

Airflow task randomly exited with return code 1 [Local Executor / PythonOperator]

To give some context, I am using Airflow 2.3.0 on Kubernetes with the Local Executor (which may sound weird, but it works for us for now) with one pod for the webserver and two for the scheduler. I have a DAG consisting of a single task…
Nicolaz
  • 149
  • 1
  • 8
0
votes
1 answer

Automatically remove dag from airflow UI not present in dagbag

Scenario I have a python file which creates multiple dags(Dynamic dag). This file fetches some data from an API and say 100 dags are created based on 100 rows from the API response. Issue When the API response changes, say now 90 rows are coming…
bha159
  • 211
  • 3
  • 14
1 2 3
…
45 46