Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.
Questions tagged [airflow-2.x]
690 questions
3
votes
1 answer
Airflow 2.0 Issues : Too many airflow supervisor tasks
I installed airflow 2.0 using docker swarm and Celery Executor.
After 1 week, celery workers memory is overflowing by airflow task supervisor (screenshot attached)
Anyone faced such issues ? Any suggestions ?
Ganesh
- 677
- 8
- 11
3
votes
2 answers
Airflow 2 - debugging why dag is not loading
On Airflow 2 my dag is not showing on the UI, and I'm getting DAG Import Errors (...) for it.
The error message is insufficient for me to debug (it's a custom operator, with a lot of custom logic - so I don't want to get into details of the error…
Grzegorz Skibinski
- 12,624
- 2
- 11
- 34
3
votes
1 answer
Where is the adhoc request option in Airflow 2.0.1?
I do not see an adhoc request in the dropdown within airflow 2.0.1. Does anyone have info on this? Was it removed from 2.0+? Any help will be greatly appreciated, I used this a lot.
RussBuss
- 33
- 2
3
votes
1 answer
Defining complex workflow dependency in airflow 2.0 taskflow API
Let's say I have the follow dummy DAG defined as below:
@dag(default_args=default_args,
schedule_interval=None,
start_date=days_ago(2))
def airflow_taskflow_api_dag():
cur_day = '2020-01-01'
@task()
def A(current_day: str):
…
FriedSquirrel
- 71
- 4
3
votes
2 answers
Airflow 2.0 Docker setup
Recently been trying to learn Airflow, but a majority of resources online depended on this repo https://github.com/puckel/docker-airflow which unfortunately has been removed.
I am not familiar with docker so I'm just trying to set up locally and…
kjay
- 383
- 3
- 16
2
votes
1 answer
Trigger the Cloud composer dag run manually from java code using a client
I have a cloud composer dag which has scheduler property set to none and needs to triggered.
I have uploaded my dag code to the cloud compose gcs folder and tried to trigger the from my local using my local gcloud credentials as suggested in the…
Sasirekha MSVL
- 113
- 2
- 12
2
votes
1 answer
Import "airflow" could not be resolved
VSCode is not able to import airflow modules.
Python == 3.10.6,
apache-airflow == 2.5.3
Airflow server is working fine, the DAGs can be created, but pylint and pylance are saying that airflow can not be imported.
Airflow is configured without…
Aidos Kenessov
- 21
- 2
2
votes
0 answers
Airflow - How to handle tasks relationship for the tasks inside dynamic task group mapping
I have implemented dynamic task group mapping with a Python operator and a deferrable operator inside the task group.
I got stuck with controlling the relationship between mapped instance value passed during runtime i.e when the deferrable operator…
saravana kumar
- 255
- 1
- 3
- 10
2
votes
0 answers
In Airflow GUI - ModuleNotFoundError: No module named 'airflow.providers.cncf
Apache Airflow version
2.5.2
What happened
Not able to create new (Admin - > Connection -> When I press + sign it gives below error regarding airflow.providers.cncf
Error:
Ooops!
Something bad has happened.
Airflow is used by many users, and it is…
Austin Jackson
- 153
- 6
2
votes
0 answers
Unable to find Airflow.cfg file
I have set up the Airflow environment on AWS EKS using the official doc
https://airflow.apache.org/docs/helm-chart/stable/index.html
The pods are up and running:
From pods description I can see that there is supposed to be a mount for…
shubham
- 182
- 1
- 1
- 8
2
votes
1 answer
How to remove a downstream or upstream task dependency in Airflow
Assuming we have the two following Airflow tasks in a DAG,
from airflow.operators.dummy import DummyOperator
t1 = DummyOperator(task_id='dummy_1')
t2 = DummyOperator(task_id='dummy_2')
we can specify dependencies as:
# Option A
t1 >> t2
# Option…
Tokyo
- 753
- 1
- 10
- 25
2
votes
2 answers
Airflow - execute X dynamic tasks with max 4 tasks in parallel
As a step one in my dag, i am trying to get a list of items from some source, say idList with count of say 100.
Is it possible in Airflow to process all the 100 items in idList, with a task concurrency of 4 maximum? (4 at a time) After one task is…
Jerald Baker
- 1,121
- 1
- 12
- 48
2
votes
2 answers
Multiple inheritance using `BaseBranchOperator` in Airflow
Can one use multiple inheritance using BaseBranchOperator in Airflow?
I want to define an operator like:
from airflow.models import BaseOperator
from airflow.operators.branch import BaseBranchOperator
class MyOperator(BaseOperator,…
cavalcantelucas
- 1,362
- 3
- 12
- 34
2
votes
0 answers
How can I set value of variable max_active_tis_per_dag at run time for a task in Airflow 2.3.x?
I am currently trying to make the parallel execution of Apache Airflow 2.3.x dynamic task.
When executing below DAG, let say 100 dynamic "sanitize_payment" task is created in "sanitize_payment_group". I have set value of "max_active_tis_per_dag" 16…
Pratyush
- 21
- 2
2
votes
0 answers
Process a large file using Apache Airflow Task Groups
I need to process a zip file(that contains a text file) using task groups in airflow. No. of lines can vary from 1 to 50 Million. I want to read the text file in the zip file process each line and write the processed line to another text file, zip…
PROTOCOL
- 371
- 9
- 17