Questions tagged [airflow-taskflow]
67 questions
0
votes
1 answer
Restrict/Exclude specific logs from Airflow to Datadog
We are observing Airflow is sending large amount of logs to Datadog and we want to restrict/Reduce these logs by excluding logs from below tasks:
pod_manager.py
base.py
base_aws.py
logging_mixin.py
Do we have any configuration settings where I can…
suresh choudhary
- 13
- 4
0
votes
1 answer
Get dag_run context in Airflow TaskFlow task
My dag is started with configuration JSON:
{"foo" : "bar"}
I have a Python operator which uses this value:
my_task = PythonOperator(
task_id="my_task",
op_kwargs={"foo": "{{ dag_run.conf['foo'] }}"},
python_callable=lambda foo:…
Aaron Brager
- 65,323
- 19
- 161
- 287
0
votes
1 answer
Apache airflow xcom for variable task id
I am currently working on a project which involves copying data from an S3 stage location to our tables managed by snowflake. However, it only works fine if I only have 1 table to copy data to(as I only need 1 task group and can therefore hard code…
Dynastywarriorlord07
- 17
- 3
0
votes
1 answer
How to retrieve nested output from XCom using taskflow syntax in Airflow
Well, I know this seems to be possible I just don't know how. To begin with, I am using traditional operators (without @task decorator) but I am interested in XComArgs return output format from these operators that can be used in downstream tasks.…
mad_
- 8,121
- 2
- 25
- 40
0
votes
1 answer
Taskflow API in google cloud composer
I am trying to upload a DAG defined with taskflow api on google cloud composer but no matter what I do, it doesn't show up in the DAGs list, neither in airflow UI or DAGs list provided by composer.
Another strange thing is that if I upload the file…
razvanv
- 53
- 6
0
votes
1 answer
Airflow - Task flow, Dynamic Task Mapping with multiple output
Let's say I have an Airflow (2.3) DAG that looks like this:
@task
def retrieve_ingest_setup_queries():
settings = Variable.get("athena_settings", deserialize_json=True)
# settings = {'drops': ["DROP TABLE my_table", "DROP TABLE my_table2"],
…
Flo
- 377
- 2
- 15
0
votes
1 answer
How to call a function from inside an airflow task?
with DAG(
"test_dag_venv",
default_args=default_args,
description='Dag to test venv',
schedule_interval="@once",
start_date=datetime(2022, 1, 6, 10, 45),
tags=['testing'],
concurrency=1,
is_paused_upon_creation=True,
…
raaj
- 403
- 1
- 5
- 17
0
votes
1 answer
Doesn't the airflow ecosystem share a single root logger?
I am trying to ship all airflow logs to kafka by attaching a new handler to the root logger, but not all logs are being published. Do I need to configure something else here?
This is what I'm doing:
custom_log_config.py
LOGGING_CONFIG =…
Aman
- 115
- 1
- 10
0
votes
1 answer
How to pass op_kwargs to Airflow TaskFlow DAG and Task?
Say I have a simple TaskFlow style DAG.
from datetime import datetime
from airflow.decorators import dag, task
from typing import Dict
@dag( start_date=datetime.now(),
schedule_interval='@once',
catchup=False)
def…
BenP
- 825
- 1
- 10
- 30
0
votes
1 answer
Explicitly set dependency TaskFlowAPI
Suppose I have a dag as follows:
def my_dag():
@task
def task_a():
# Some code here
return
@task
def task_b():
# Some code here
return
task_a()
task_b()
my_dag_instance = my_dag()
I would…
Diana Vazquez Romo
- 152
- 1
- 1
- 11
0
votes
2 answers
Logging 'logical_date' or 'ds' in Airflow task
I am using Airflow and writing my DAG with Task Flow API. This is an example of one: https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html
In one of my task functions, I want to log the logical date of my dag. Thus, inside…
Diana Vazquez Romo
- 152
- 1
- 1
- 11
0
votes
1 answer
Ariflow DAG using Task flow
I recently started using Apache Airflow and after using conventional way of creating DAGs and tasks, decided to use Taskflow API. However, I ran into some issues, so here are my questions.
Conventional way:
with DAG(
dag_id="abc_test_dag",
…
Richard
- 1
- 1
0
votes
1 answer
Airflow and TaskFlow API first steps: 'NoneType' object is not iterable
I am trying to understand how TaskFlow API works and I am starting from this basic DAG:
def _push_argument(ti):
ti.xcom_push(key='p1', value={'param1': 'uno', 'param2': 'due'})
def _print_data(param1, param2):
print(param1)
…
afmulone
- 177
- 1
- 1
- 8
0
votes
2 answers
Airflow 2.0: Encapsulating DAG in class using Taskflow API
I have pipelines where the mechanics are always the same, a sequence of two tasks.
So I try to abstract the construction of it through a parent abstract class (using TaskFlow API):
from abc import ABC, abstractmethod
from airflow.decorators import…
Nicolaz
- 149
- 1
- 8
0
votes
2 answers
Airflow 2: Check if a task still exists or not with task_id
we have created a task for sensor operation, but the task name will be dynamically updated. i.e., f"{table_name}_s3_exists". We have a scenario where we have to check a table's location twice, but if the task is still present, we don't have to…
Santhosh
- 71
- 1
- 5