Stack Overflow
Stack Overflow

Questions tagged [airflow-taskflow]

67 questions
1
vote
0 answers

How to create TaskFlow DAGs dynamically in AirFlow 2.0?

I have a parameterized DAG and I want to programmatically create DAGs instances based on this DAG. In traditional Airflow model, I can achieve this easily using a loop: # Code sample from:…
lifanxi
  • 11
  • 1
0
votes
0 answers

Override/update one of the params through a file in Airflow

How to update the params on the fly? I want to update 'input_file' param to the path present in file 'temp_file.txt' import datetime as dt from airflow import DAG from airflow.providers.ssh.operators.ssh import SSHOperator from…
tanviz1992
  • 1
  • 1
0
votes
0 answers

Airflow dynamic KubernetesPodOperator does not handle zero-length maps

I have two dynamically generated tasks task_1 and task_2. task_2 depends on task_1. Both are instances of KubernetesPodOperator. If an empty map (0 length) is passed to task_2, instead of skipping task_2 per the documentation, task_2 is put in a…
Hui
  • 97
  • 7
0
votes
0 answers

Airflow Dynamic TaskGroup Mapping - How to use return value from task that is dependent on others?

Hey so I am using Airflow 2.5.3 and Dynamic TaskGroup Mapping so I can iterate over rows in a table and use the values in those rows as parameters in this group of tasks. That is all working fine, and I am getting close to completing what I need to…
Jake Wesolowski
  • 1
  • 1
0
votes
0 answers

AIrflow db init not picking up external dependencies

Problem: running airflow db init in my virtualenv gives the following error which I can see is installed with a pip freeze ❯ airflow db init DB: sqlite:////Users/.../airflow.db [2023-07-24T23:03:08.958+0100] {migration.py:213} INFO - Context impl…
John Phillips
  • 1
  • 2
0
votes
1 answer

Airflow Dag isn't completing its execution. [heartbeat] error

I am trying to take baby steps with Airflow. I was able to test the task successfully via VS Terminal. However, when I trigger the Dag trough UI it just stay in 'running' stage forever. When I forcefully mark as failed and review logs. I see it as…
Sri_Dev
  • 1
  • 2
0
votes
1 answer

How to run downstream dynamic tasks as soon as dependencies are fullfilled?

I have an airflow DAG using 3 Tasks. Task 1 is generating a map, based on which I'm branching out downstream tasks. The pipeline loooks like this: Task 1 --> Task 2a --> Task 3a | |---> Task 2b --> Task 3b My problem: If I have a failure…
radschapur
  • 445
  • 1
  • 3
  • 13
0
votes
1 answer

Can you put retries on an individual TaskFlow @task?

All the examples of retries on tasks in the Airflow docs are on things like BashOperator. Manual tests on version 2.5.1 show that this doesn't work on Taskflow: @task def test_retries(retries=2): raise ValueError("I failed, please…
Noumenon
  • 5,099
  • 4
  • 53
  • 73
0
votes
1 answer

Create Dynamic Airflow tasks with separate dependencies

I want to generate multiple Airflow sensors/operators in a loop, but I want to be able to access them one-by-one, as they have different dependencies. For example, task1 has a dependency for operator1, operator2, and operator3; while task2 has a…
Olivér Horváth
  • 11
  • 2
0
votes
1 answer

How can i get all dags details from airflow rest api

i have more dags (say 150dags) in my airflow account. My airflow version = 2.4.0 im using url and query parameter http://localhost:8081/api/v1/dags?limit=1000&offset=0 But my response consists only 100 records(dags). i think query parameters are not…
vbala vbala
  • 15
  • 3
0
votes
1 answer

How to get the status of DAG_RUN_STATUS for a DAG in airflow

How to get the execution status of overall DAG, which can be used to trigger the mail. def on_success_dag(context): dag = context.get("task_instance").dag_id state=context.get("task_instance").state SUBJECT="" MSG = """ …
Austin Jackson
  • 153
  • 6
0
votes
1 answer

Airflow Sensor incomplete files list

I'm really curious to know, how airflow sensors work when a file is not completely copied/loaded to a target data storage. As an example: we have a filesystem, and sensor checks files within it. We are copying a large file to folder and it really…
Data Therapist
  • 3
  • 3
0
votes
1 answer

airflow timeout error for SSH operator while executing a docker image

Getting error of time-out for the task of SSH while executing the docker image and its commands within it. [2023-06-14, 16:07:03 UTC] {taskinstance.py:1776} ERROR - Task failed with exception Traceback (most recent call last): File…
Austin Jackson
  • 153
  • 6
0
votes
0 answers

Airflow - Import multiple CSV files to Postgres

I want to construct a pipeline that will be activated when multiple CSV files are deposited in a specific folder. Each CSV file will be read and then saved to a Postgres database. Following are my questions and my code: How can I determine the name…
Grigoris Papapostolou
  • 1
0
votes
0 answers

Airflow Dynamic Mapped Task: MappedArgument is not JSON serializable

Here's the problem I am trying to solve and run into issues with XCom Args The general idea is to have a pipeline that's extracting data from postgres databases and dumping them into BigQuery. The pipeline is composed of the DAG linked together via…
QuantumLicht
  • 2,103
  • 3
  • 23
  • 32
1
2
3 4 5