Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.
Questions tagged [airflow-2.x]
690 questions
0
votes
1 answer
Airflow Gitsync Not syncing Dags - Community Helm Chart
I am attempting to use the Gitsync option to Load Dags with the Community Airflow Helm Chart. It appears to be syncing in the init container (dags-git-clone)
I0824 15:21:52.114912 14 main.go:473] "level"=0 "msg"="starting up" "pid"=14…
ColeGulledge
- 393
- 1
- 2
- 12
0
votes
2 answers
Newly installed Airflow providers don't show in Web UI
I'm using MWAA and when I add new providers via requirements.txt they do not show in the "Airflow - Admin > Providers" UI, while the default providers installed on the MWAA instance show just fine.
An example:
I add…
theNewfelll
- 19
- 4
0
votes
1 answer
BigQueryGetDataOperator project ID
I read the document about BigQueryGetDataOperator and used it. The problem is I didn't know have to pass my select projectID to this operator. So this returns non exist table.
It default use: project_a.dataset_b.table_c.
With I would like to change…
Trong Van
- 374
- 1
- 13
0
votes
2 answers
Airflow:SnowflakeOperator:Assign values from previous statement in SQL query
Snowflake:Assign values from previous statement in SQL query
Requirement: Assign values from the previous statement to the next statement in SQL query , as I run the query in SnowflakeOperator in Airflow
SQL:
BEGIN
app = 'abc';
env…
Kar
- 790
- 13
- 36
0
votes
1 answer
Airflow bashoperator CommandNotFoundError
I wanto run this dag file in airflow 2. But it isn't working with error below. I can run this same command in terminal. But it isn't working dag environment of airflow.
Error
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-08-16T02:29:00+00:00
[2022-08-16,…
verystrongjoe
- 3,831
- 9
- 35
- 66
0
votes
0 answers
Does Spark Config Override Spark Session
I am using Airflow to create a DAG to run a spark job. In the DAG I am setting the spark_conf (executor memory, driver memory, and maxResultSize) for my use case.
I received a GC memory error so, as I was reviewing the python code I noticed a spark…
Prof. Falken
- 499
- 6
- 21
0
votes
1 answer
Email alerts on failure in Airflow without turning on "Allow less secure apps"
To send email failure alerts, Airflow needed an SMTP username and password for authentication. I am using google mail for that but google treats all apps that use username and password as less secure. So to make it work, I need to enable "Allow less…
hemant A
- 185
- 14
0
votes
1 answer
Create a google bigquery connection from Airflow UI (Dockerized)
I am running an Airflow instance using Docker. I am able to access the Airflow UI using http://localhost:8080/. Also able to execute a sample dag using PythonOperator. Using PythonOperator I am able to query a big query table on GCP environment. The…
sandeep
- 3,061
- 11
- 35
- 54
0
votes
2 answers
All DAGs broken after MWAA update from 2.0.2 to 2.2.2
I am getting the following errors in AWS MWAA UI after I updated from 2.0.2 to 2.2.2
I have exhaustively searched for more details on these errors to no avail
from airflow.providers.slack.operators.slack_webhook import…
ab_padfoot
- 63
- 1
- 10
0
votes
2 answers
How to avoid dynamic execution of expression in dag parameter at Airflow?
I'm using a parameter that is the timestamp in a set of tasks:
default_dag_args = {'arg1': 'arg1-value',
'arg2': 'arg2-value',
'now': datetime.now()}
I would like that the now parameter would have the same…
Matheus Oliveira
- 587
- 3
- 10
- 33
0
votes
0 answers
how to solve ModuleNotFoundError: No module named 'utilities' in airflow 2.3.3?
I have installed new utilities-package but also the issue is not solved and could not get any new solution from the web.
error:
Broken DAG: [/data/airflow/dags/DA.py] Traceback (most recent call last):
File "", line…
0
votes
3 answers
ModuleNotFoundError while importing Airflow DAG
I am currently working with a local installation of Airflow on Ubuntu, and this works perfectly for simple DAGs.
I now have a DAG that has dependencies on other modules, and I am stuck with a ModuleNotFound error that stops me from importing the…
hobscrk
- 17
- 1
- 7
0
votes
1 answer
How to only run certain operator when dag conf value exist
def skip_update_job_pod_name(dag):
"""
:param dag: Airflow DAG
:return: Dummy operator to skip update pod name
"""
return DummyOperator(task_id="skip_update_job_pod_name", dag=dag)
def update_pod_name_branch_operator(dag: DAG,…
WOWpopo
- 15
- 4
0
votes
2 answers
How can I add new Conn Types (MYSQL) to Airflow 2.0 Astronomer?
I'm learning Apache Airflow 2.0 astronomer and I was creating a new "Connection" but in "Conn Type" there are just a few options, for example I want to add a MySQL connection but there isn't that option. enter image description here
Agus Mahari
- 1
- 2
0
votes
1 answer
How to properly add operators based on array result retreived from another operator in Apache Airflow?
I need to first read from BigQuery to get a list of SQL strings stored in a specific table then loop through the string array and run against BigQuery again to fix things. For running those SQLs I want to run them parallel to speed things up.
My…
tom10271
- 4,222
- 5
- 33
- 62