Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.
Questions tagged [airflow-2.x]
690 questions
1
vote
2 answers
datetime.strptime unable to render xcom variable
I have this simple dag with two tasks, one which returns a datetime as str and another which uses that to run TimeSensorAsync deferred operator.
@task
def get_scheduled_upgrade_time(data: dict) -> str:
return str(datetime.utcnow() +…
sharpnife
- 198
- 12
1
vote
0 answers
How to identify airflow dependent DAGS and Tasks through airflow rest api
i have some dependent dags (parent and child dags). How to i identify those dags are dependent ,
How to classify whether dag is dependent or not.
im using airflow version 2.4.0
How to get dependent dags details through airflow rest api.
Ref : How to…
vbala vbala
- 15
- 3
1
vote
1 answer
Is there any way having restrictions on dags by their owners upon roles
I have some dags owner by user 'abc' and I want to this user a role with permissions to show just those dags on airflow GUI which are owned by user 'abc' and this user can pause or unpause or run its own owned dag. Just to clear here that I cannot…
Usman_Ahmed
- 67
- 1
- 9
1
vote
1 answer
Cannot find AwaitableTriggerDagRunOperator anymore for Airflow [Python]
I'm working on a python project using Airflow. In the project there is no requirements.txt file so I simply installed the latest version of the libraries by putting their name inside a requirement.txt file and I've been trying to make it work.
The…
Gerardo Zinno
- 1,518
- 1
- 13
- 35
1
vote
0 answers
Apache Airflow: Clean a task with a config
I'm using Apache Airflow 2, I can re-run the whole dag using a config:
However sometimes I want to re-run only one task and not the whole dag with a config also, for now I can clear the task, but I'm not able to add a config, the task I want to…
svg_af_2
- 75
- 5
1
vote
1 answer
How to inject inlets and outlets parameters in Airflow PythonOperator executable function
I'd like to automatically set inlets and outlets parameters in executable function inside PythonOperator.
But, it seems to me that it doesn't work while it should. You can find the code snippet below:
from datahub_provider import entities
def…
Rodion Proskuriakov
- 11
- 2
1
vote
2 answers
Why is my Airflow hanging up if I send a http request inside a task?
System: MacOS Apple M1 (local machine)
Airflow: 2.5.3
Executor: Local with Postgres Database
I am trying to implement some externally triggered workflows which load data from our REST APIs. I am using the Python Operator to run the code and trigger…
Alexandr Sarioglo
- 189
- 6
1
vote
0 answers
Why airflow marks kubernetes pod operator tasks in airflow as failed on the UI while the containers are still running on k8s?
I am having a for loop in my DAG which creates some tasks using kubernetes pod operator. These tasks run in parallel and there are enough resources available on the k8s nodes to let them run in parallel. These containers share an EFS volume as a PVC…
Naxi
- 1,504
- 5
- 33
- 72
1
vote
1 answer
How pull xcom value between task and taskgroup?
I have a requirement where I have to do xcom pull to fet few details from previous task.I tried many options but did not work in Airflow 2.2.3.
Here is a sample code example
from airflow.operators.python_operator import PythonOperator
from…
Kumar Padhy
- 91
- 9
1
vote
1 answer
dbt Cloud airflow conn_id from secret manager breaking parsing
I'm using aws secret manager to store my connections in airflow.
So when creating the dbt connection string in the secret , i followed this…
callmedivs
- 33
- 5
1
vote
1 answer
Why does the default Cloud Composer 2 have parallelism set to 0?
I am going through my newly created Cloud Composer 2 environment configuration. Previously I only used the version 1 and I still don't feel familiar with some aspects of Cloud Composer 2. While looking at the configuration, I realised that there is…
BloodthirstyPlatypus
- 79
- 6
1
vote
0 answers
Using Airflow, How to handle asynchronous API calls with Airflow Tasks?
I'm quite new to Airflow, I need to make asynchronous POST API calls to start the execution of the external service. And Need to make GET API calls to check the status of the execution and have to make that call until the execution gets completed.
I…
saravana kumar
- 255
- 1
- 3
- 10
1
vote
1 answer
Airflow Jinja renders list of integer strings as int tuple
I currently have the following DAG structure:
with airflow.DAG(
'example_dag',
render_template_as_native_obj=True,
params={
'array': ['1', '2', '3'],
},
) as dag:
where I use p like this:
dummy_job = job_operator.JobOperator(
…
Tony Liao
- 11
- 1
1
vote
0 answers
Access XCOM values outside operator
I need to access XCOM values outside the operator, it can be done using variables, but i want to avoid it.
I am pushing the xcom values from one operator and then a funcction to fetch xcom values using getone/getmany function, but it is returning…
Ankit Gupta
- 11
- 2
1
vote
0 answers
Airflow retries long running task multiple times while the task is still running
I have a long running task where it loops through calling some REST endpoint to get data maybe hundreds of times and could take up to 1 hour.
While the task is still running, I see there are multiple attempts or retries, even though I specifically…
nismoh
- 53
- 1
- 7