Stack Overflow
Stack Overflow

Questions tagged [airflow-taskflow]

67 questions
0
votes
0 answers

Airflow Dynamic Task Mapping showing "No Data found" for Mapped Task

I have airflow version 2.6.0 and python version 3.9.2. I am following the official tutorial of Dynamic Task Mapping and I am using the example code for a dag: from __future__ import annotations from datetime import datetime from airflow import…
Gustavo Martins Pinheiro
  • 1
  • 1
0
votes
1 answer

Airflow sensor unable to access context variable

I'm trying to build a sensor that reads the dag parameters (that you can change when you trigger dag with config) to know how long to wait. from airflow.decorators import dag, task, task_group from datetime import date, datetime, timedelta import…
Thomas
  • 543
  • 5
  • 11
0
votes
1 answer

How to mark BashOperator in airflow success

I am using bashOperator to execute ETL script on GCP compute engine and some files can take more than 10hrs to complete. Since I am using compute engine to execute how can I set bashoperator task to success and make the dag run to success so that I…
avinash reddy
  • 25
  • 4
0
votes
1 answer

GCSToGCSOperator not moving files in same bucket

I am trying to move files from my "new" folder in my-bucket to "test" folder in the same bucket using GCSToGCSOperator, I am passing a list of files to the source bucket but when I run dag I don't see files moving. I used xcom push to get list of…
avinash reddy
  • 25
  • 4
0
votes
1 answer

ModuleNotFoundError: No module named 'my_package'. Airflow not find my_package in its Python environment

I set up my DAG in docker container as follows: with DAG( "my_dag", default_args=dag_config, schedule="@weekly", ) as dag: config_env = BashOperator( task_id="config_env", bash_command="cd…
panday1995
  • 11
  • 3
0
votes
0 answers

Airflow no module found error 'airflow.providers.airbyte' on UI

Broken DAG: [/usr/local/airflow/dags/dag_airbyte_example.py] Traceback (most recent call last): File "", line 241, in _call_with_frames_removed File "/usr/local/airflow/dags/dag_airbyte_example.py", line 3, in…
Shruti Kulkarni
  • 11
  • 3
0
votes
1 answer

How to print and store BashCommand output in Airflow?

I have a DAG that executes multiple commands that are stored in a JSON file (most of them are python scripts that runs with arguments), the structure of the DAG is similar to this: def execute_command_and_store_logs_in_mongodb(): blablabla def…
Tadeo
  • 1
  • 1
0
votes
1 answer

How to best mix-and-match returning/non-returning tasks on Airflow Taskflow API?

Note: All examples below seek a one-line serial execution. Dependencies on Taskflow API can be easily made serial if they don't return data that is used afterwards: t1() >> t2() If the task T1 returning a value and task T2 using it, you can also…
xmar
  • 1,729
  • 20
  • 48
0
votes
1 answer

AIRFLOW Redshift Hook

Airflow novice here Setup env: MWAA db: Redshift Question: I am trying to write a task to loop through a list, use the list value in SQL and lastly use an if statement to determine appropriate action based on the results. Something like this but…
AIViz
  • 82
  • 9
0
votes
1 answer

Airflow v2.3.4: Make all tasks in a DAG run at the same time

How can a define the parameters for airflow KubernetesPodOperator make all tasks in a DAG run at the same time. In my image below you can see that some tasks are in grey "scheduled", I want them to run all at the same time green, also make it NOT…
The Dan
  • 1,408
  • 6
  • 16
  • 41
0
votes
2 answers

airflow.exceptions.AirflowException: 'branch_task_ids' must contain only valid task_ids

I have a dag which contains 1 custom task, 1 @task.branch task decorator and 1 taskgroup, inside the taskgroup I have multiple tasks that need to be triggered sequentially depending on the outcome of the @task.branch. PROCESS_BATCH_data_FILE =…
Mgoga
  • 3
  • 2
0
votes
1 answer

Airflow TaskFlow API connect with filesensor

I'm learning Airflow TaskFlow API and now I struggle with following problem: I'm trying to make dependencies between FileSensor() and @task and I want to run my download() task after successful result of my FileSensor @dag(): def my_dag(): …
Illia Kaltovich
  • 15
  • 4
0
votes
0 answers

Airflow: using DAG manually informed params outside of task but inside of DAG

I defined a DAG which has two params date_start and date_end, which can be manually informed when DAG is manually triggered using config. Given this dates, before passing them to triggered DAGs, I want to verify if they have the correct format and…
Cristian Ispan
  • 571
  • 2
  • 5
  • 23
0
votes
0 answers

Is there any way to avoid failure and continue runningtasks with a set limt for airflow dynamic tasks

We have the max_map_length variable configured in airflow configuration which will make the dynamic task fail if the length of mapped tasks is more than the set limit eg: max_map_length = 1024 , if the task length is more than , the task will fail…
Boss Boss
  • 3
  • 1
  • 5
0
votes
0 answers

Use dic output from a PythonOperator to iterate another operator (dbt) in Airflow

I want to first get a dic from one operator, and then iterate the result and run another operator. The dictionary contains dataset and tables which contain words like "daily" "weekly" or "monthly". The script is as follows: @task def…
RubTest
  • 21
  • 3
1 2
3
4 5