Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.
Questions tagged [airflow-2.x]
690 questions
3
votes
1 answer
on_failure_callback triggered multiple times
I want to publish SINGLE Kafka message in case of airflow PARALLEL task failures. my airflow dags are similar to below.
from datetime import datetime, timedelta
from airflow.models import Variable
from airflow import DAG
from airflow.operators.dummy…
user1992
- 169
- 1
- 15
3
votes
1 answer
How to get Airflow Docker ExternalPythonOperator working with in a python venv?
Situation
Since 2022 Sept 19 The release of Apache Airflow 2.4.0
Airflow supports ExternalPythonOperator
I have asked the main contributors as well and I should be able to add 2 python virtual environments to the base image of Airflow Docker 2.4.1…
sogu
- 2,738
- 5
- 31
- 90
3
votes
1 answer
What parameters can be passed to Airflow @task decorator?
I am using Airflow 2.4.0.
from airflow.decorators import task
from airflow import DAG
with DAG(
"hello_world",
start_date=datetime(2022, 1, 1),
schedule_interval="@daily",
catchup=False,
) as dag:
@task()
def…
Hongbo Miao
- 45,290
- 60
- 174
- 267
3
votes
0 answers
Airflow Not triggering sla_miss_callback for my case
I have followed the documentation and previous stackoverflow links eg : Airflow sla_miss_callback function not triggering
Still I am not able to trigger sla_miss_callback for my case.
I have a top level DAG(DAG X1) that has scheduler interval of : 0…
Mayank Kataruka
- 395
- 1
- 2
- 11
3
votes
1 answer
Airflow External Task Sensor with unscheduled upstream DAG
We use airflow in a hybrid ETL system. By this I mean that some of our DAGs are not scheduled but externally triggered using the Airflow API.
We are trying to do the following: Have a sensor in a scheduled DAG (DAG1) that senses that a task inside…
gontxomde
- 132
- 9
3
votes
3 answers
Airflow:Docker-Compose:"You are running pip as root Please use user to run pip"
Requirement: To run local Airflow using official docker-compose with Airflow version 2.3.0
Issue: "You are running pip as root Please use user to run pip"
Airflow version 2.3.0
OS - macOS
I was able to run fine with 2.2.4 but not with 2.3.0
Also ran…
Karthik
- 441
- 5
- 17
3
votes
0 answers
How to connect MWAA CLI that is hosted in a public subnet
I am trying to invoke airflow Cli from my local terminal and I have followed these instruction https://dnx.solutions/how-to-use-apache-airflow-cli-with-amazon-mwaa/
and created cli_token and executed
curl \
--request POST…
Xi12
- 939
- 2
- 14
- 27
3
votes
1 answer
What is task_instance.xcom_pull in AIrflow?
I am trying to run EMR through Airflow and found example where it says
step_adder = EmrAddStepsOperator(
task_id='add_steps',
job_flow_id="{{ task_instance.xcom_pull(task_ids='create_job_flow', key='return_value') }}",
…
Xi12
- 939
- 2
- 14
- 27
3
votes
1 answer
airflow kubernetes default setup ERROR: relation "log" does not exist at character 13
I deployed the default helm chart for airflow 2.
The postgres pod is reporting an error:
ERROR: relation "log" does not exist at character 13
This appears after this statement:
INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner,…
vbp13
- 1,040
- 1
- 10
- 20
3
votes
1 answer
Dynamically generate multiple tasks based on output dictionary from task in Airflow
I have a task in which the output is a dictionary with a list value in each key
@task(task_id="gen_dict")
def generate_dict():
...
return output_dict # output look like this {"A" : ["aa","bb", "cc"], "B" : ["dd","ee", "ff"]}
# my dag (Not…
emp
- 602
- 3
- 11
- 22
3
votes
1 answer
Airflow setting up Google Cloud connection
I am setting up a google cloud connection type in Airflows UI. I am running Airlow 2.2 in docker locally.
I am adding what I understand are the required items for using a service account. However, on testing, I get a 400 Bad Request.
{ "detail":…
airportmarc
- 35
- 2
3
votes
2 answers
Error in airflow 2 even though my task's have actually completed? ERROR - Could not serialize the XCom value into JSON
Hi All so my dag actully runs fine, all the outputs are working but airflow's UI does not change to succes and fails due to the following issue. Reading online and i have come across these 2:
do_xcom_push=False
and that Xcom_push will be deprecated…
Mizanur Choudhury
- 324
- 2
- 5
- 16
3
votes
1 answer
Cannot setup a MySQL Backend for Airflow LocalExecutor
I need to run dags in parallel but do not need significant scaling, so LocalExecutor can do the job just fine. I looked through the Airflow docs and first created a MySQL database:
CREATE DATABASE airflow_db CHARACTER SET utf8;
CREATE USER …
Aaron Gonzalez
- 86
- 11
3
votes
2 answers
How to use jinja template in Airflow MySQL Operator
I am currently running this query in Airflow's MysQLOperator.
How can I replace region, s3 bucket with parameters using Jinja template?
Airflow version: 2.0.2
Python: 3.7
sql = """SELECT * FROM test
INTO OUTFILE S3…
Yohei Onishi
- 1,362
- 1
- 19
- 42
3
votes
0 answers
Airflow tasks are not running
I deployed webserver, scheduler, flower and worker on ecs fargate using docker image. I am using latest airflow version 2.1.2. when i am running dag, the worker node is throwing error saying the below:
[2021-08-13 11:38:45,323:…
Mani
- 85
- 10