Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.
Questions tagged [airflow-2.x]
690 questions
0
votes
1 answer
branch trigger rule all_success or none_failed does not work, task gets executed even if it is not supposed to
I have an airflow DAG as below
task1 -> task2 -> [task3_1, task3_2]
task3_1 -> task4_1 -> task5_1
task3_2 -> task4_2 -> task5_2
[task5_1,task5_2] -> task_6 -> task_7 -> task_8
Tasks 1,2,6,7,8 are linear with 2 parallel jobs for 3,4,5 in between.
I…
Venkatesh Gotimukul
- 741
- 1
- 9
- 30
0
votes
1 answer
Airflow 2 taskflow logging
I'm creating a fairly complex task using the task-flow API (@task annotation) and I need to write some logs which will help me troubleshooting the runs during development and in the future.
I've researched the subject and found nothing in the…
cosenmarco
- 125
- 9
0
votes
2 answers
Running Airflow DAGs in a separate virtual environment than the original base environment of Airflow
I am running Airflow in a separate virtual env and running a couple of data quality DAGs with specific requirements. Effectively I want to run the DAGs in their own virtual envs rather than cluttering the base Airflow…
Saradindu Sengupta
- 11
- 1
- 5
0
votes
0 answers
airflow TypeError: cannot pickle 'pyodbc.Cursor' object
{python.py:175} INFO - Done. Returned value was:
[2022-03-18, 17:56:17 UTC] {taskinstance.py:1700} ERROR - Task failed with exception
Traceback (most recent call last):
File…
vinsent paramanantham
- 953
- 3
- 15
- 34
0
votes
1 answer
Airflow Worker is not working when using CeleryExecutor
I'm trying to migrate from LocalExecutor to CeleryExecutor in Airflow 2.1.3 using Docker with Redis. I made separate containers for webserver, scheduler, worker, redis and database. Problem: tasks are queued but not…
Stepan Pryn
- 1
- 1
0
votes
1 answer
In the official helm chart, how do you specify the subpath for the dags when using persistence volume claim?
It was working before with the previous chart which I think was the community version. Since the official helm has been created/launched, I want to migrate to the official one. I did not see that you can specify a subPath. This is for Airflow…
alltej
- 6,787
- 10
- 46
- 87
0
votes
1 answer
Airflow: Get the status of the prior run for a task
I'm working with Airflow 2.1.4 and looking to find the status of the prior task run (Task Run, not Task Instance and not Dag Run).
I.e., DAG MorningWorkflow runs a 9:00am, and task ConditionalTask is in that dag. There is some precondition logic…
Kevin Crouse
- 43
- 1
- 7
0
votes
1 answer
Airflow - Dynamic Tasks and Downstream Dependencies
I inherited the following dag which is running on AWS MWAA v2.2.2
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.utils.dates import days_ago
from airflow.providers.snowflake.operators.snowflake import…
Damien
- 4,081
- 12
- 75
- 126
0
votes
0 answers
Prettify SQL query in Airflow2 rendered template when using thew new BigQuery operators
Our team is currently migrating dags to Airflow 2. One of the changes we're applying is moving from the deprecated BQ ops to the new ones, for example, BigQueryInsertJobOperator
The problem is, with these new operators, the query is no longer passed…
Dasph
- 420
- 2
- 15
0
votes
1 answer
Airflow - Unable to Create Table in Snowflake
I am running Airflow v2.2.2 using the MWAA service on AWS
I have the following DAG
import airflow
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
import time
from datetime import timedelta
import…
Damien
- 4,081
- 12
- 75
- 126
0
votes
1 answer
Installed AirFLow 2.2.3 but getting error AttributeError: task decorator 'docker' not found
Just trying to run the example in:
https://github.com/apache/airflow/blob/main/airflow/example_dags/tutorial_etl_dag.py
But basically getting this (removed the try/except at the and and some comments, so line numbers might not match).
Is there…
Sonny D
- 11
- 3
0
votes
1 answer
Passing params through jinja templates in SubDagOperator in Airflow 2
Airflow 2: I have pushed an xcom from taskA and I am pulling that xcom within subdag taskB. I have been unable to pull the necessary xcom. I was wondering how one would do this.
I was able to pull the necessary xcom with same piece of code in…
codninja0908
- 497
- 8
- 29
0
votes
1 answer
How to change max duration or refresh AWS session in Airflow 2.0
Can we increase session duration or refresh session for AWS in Apache Airflow v 2.0.2 when we use assume role like in AWS connection extra?
{
"role_arn": "arn:aws:iam::xxx:role/airflow-connection-role",
"region_name": "ap-southeast-1"
}
I…
Yohei Onishi
- 1,362
- 1
- 19
- 42
0
votes
1 answer
How to know what caused a simple DAG in Apache Airflow to fail?
I am new to Apache Airflow.
I wanted to create a simple DAG with a single Task that launches a python script in a virtual environment:
from airflow import DAG
from datetime import datetime, timedelta
from airflow.operators.bash import…
Tommaso Bendinelli
- 531
- 7
- 19
0
votes
1 answer
Airflow Not Sending Errors to Sentry Automatically
I am having trouble configuring Airflow to automatically send error messages to Sentry.
I have set sentry_on and sentry_dsn in environment variables (AIRFLOW__SENTRY__SENTRY_ON and AIRFLOW__SENTRY__SENTRY_DSN), and these appear in the Airflow GUI…
tlskr
- 71
- 4