Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.
Questions tagged [airflow-2.x]
690 questions
0
votes
0 answers
Apache Airflow : How to pass a non templated argument to a template field in Airflow 2.0
I am designing a custom sensor, MySensor, in Airflow 2 which i am creating by overriding the BaseSensorOperator. I have a parameter called file which is passed to my constructor as a template field
template_fields = ("file",)
I have an Airflow…
sanchit08
- 119
- 1
- 7
0
votes
1 answer
Error when Creating a connecion on Airflow webUI
I'm using the demo database. But airflow is running with docker-compose.
I inserted the true information but i still have this error.
QDex
- 139
- 9
0
votes
1 answer
Airflow - create task that runs once all other tasks ran successfully
Is there a way to add a task that runs once all other tasks have run successfully in the same DAG, see below for my current DAG.
For example my current tasks run in the below order, but I want to add new_task once all of the below runs. If I leave…
KristiLuna
- 1,601
- 2
- 18
- 52
0
votes
1 answer
Skip the remaining tasks within airflow Dag if s3senors is not able to find the file
Is there a way I can skip remaining tasks within a dag if s3sensor operator is not able to find the file in s3 location.
I know that we can use shortcircuit operator to skip the task, but I am looking for a way in which I can integrate shortcircuit…
Apurv
- 1
- 1
0
votes
0 answers
Apache Airflow : How to write a custom sensor that checks for a certain condition and performs operation
I want to write a custom sensor in Airflow 2. The standard procedure is to override the poke method to check for the condition you want and terminate the task when it is reached. But what i want is for my sensor to initially perform an operation,…
sanchit08
- 119
- 1
- 7
0
votes
1 answer
Google Cloud Airflow Composer Produce Disk Cost "WHY" or "FROM WHERE"?
We have a problem. When we created airflow composer on google cloud platform. Disk price is come up in the billing report. But we didn't know "why do we have a cost or from which tool produce this cost?"
If you have any idea about this problem,…
0
votes
1 answer
How to switch between two different schemas in airflow dag?
I am struggling with something that looks very simple, but I don't know how to do that. we have 2 different schemas in our PRE env. How can I switch between 2 schemas in airflow? Imagine that I have only one dag .The same scenario in bash is to…
pm1359
- 622
- 1
- 10
- 31
0
votes
0 answers
DataflowPythonOperator in Airflow 2 failing on get 404
I am trying to run a job in Airflow 2.1.2 which executes a dataflow job. The dataflow job reads data from storage bucket and uploads it to bigquery. The dataflow_default_options in the DAG has region defined as europe-west1 however it is overridden…
ankie
- 11
- 2
0
votes
1 answer
airflow2.0 with KubernetesPodOperator: TemplateNotFound
I am using Airflow2.0 with KubernetesPodOperator want to run a command that use as a parameter a file from inside the image ran by the Operator. This is what I used:
KubernetesPodOperator(
namespace=commons.kubernetes_namespace,
…
salvob
- 1,300
- 3
- 21
- 41
0
votes
1 answer
Create user with LDAP authentification in airflow 2.1.4
We've updated airflow from 2.0.2 to 2.1.4 and we use LDAP for authentification in our "webserver_config.py", everything works fine. We're logged in as admin and checked the permissions in the admin role (all access given).
In the new airflow version…
thorstenleidl
- 423
- 3
- 8
0
votes
1 answer
Not able to start airflow webserver in background in docker
I am trying to run airflow on an anaconda3 docker image. Everything is working fine, except I am not able to run airflow webserver and scheduler in the background using:
airflow webserver -D
airflow scheduler -D
Both of them start in the foreground…
Nevin Baiju
- 305
- 3
- 18
0
votes
2 answers
Invoke Java class or method from Airflow
I want to invoke few java methods from an Airflow task(s) since I already have lots of code written in java including validation, cleansing, some business logic and I don't want to rewrite that in Python again.
I am using Airflow 2.1.0
I know there…
Shashank Gupta
- 321
- 3
- 15
0
votes
0 answers
Multiple BranchPythonOperator DAG configuration
import logging
import pandas as pd
import boto3
from datetime import datetime
from airflow import DAG, settings
from airflow.operators.python import PythonOperator, BranchPythonOperator
from airflow.operators.dummy_operator import…
coding_enthu
- 1
- 1
0
votes
1 answer
Problem defining DAG with Airflow 2 Taskflow API
I created a DAG with airflow 2 Taskflow API:
with airflow.DAG("plot", schedule_interval=None, default_args=default_args) as dag:
cf = collect_files()
upi = update_process_info(cf)
for i in range(0, max_parallel_plot_tasks):
…
opfau
- 731
- 9
- 37
0
votes
1 answer
Airflow webserver error when updated to airflow 2.0
After upgrading from Airflow 1.10.12 to Airflow 2.0.0, I get the following error when trying to run airflow webserver:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
response =…
kaxil
- 17,706
- 2
- 59
- 78