Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.
Questions tagged [airflow-2.x]
690 questions
0
votes
1 answer
Is it possible to trigger an Airflow 2 dag from an Airflow 1 dag, and viceversa?
We are doing a migration between new and old Airflow and we were trying to implement a trigger from Airflow1 to Airflow2(and vice-versa), is it possible?
user18141140
- 3
- 1
0
votes
1 answer
Airflow dags subdirectory - role permissions
I have an Airflow problem that seems very simple but I'm stuck with it.
I have a subdirectory (named student_dags) in my dags folder. That subdirectory has dags (python scripts).
In the Airflow UI I can see the dags from my dags folder as well as…
1--
- 61
- 5
0
votes
0 answers
Google cloud composer-airflow throws error: Broken DAG: File "/opt/python3.8/lib/python3.8/site-packages/google/cloud/bigquery/_pandas_helpers.py
I am trying to created a DAG in Cloud Composer. When importing, I'm getting the following error:
Broken DAG: [/home/airflow/gcs/dags/branch_name/persoflow/dags/test_dag.py] Traceback (most recent call last):
File…
codninja0908
- 497
- 8
- 29
0
votes
1 answer
Error executing Dataflow task from Cloud composer: Log file is not found
In one of our Cloud Composer/Airflow jobs, we invoke a number of DataflowTemplateOperator jobs (JDBC to BigQuery) to bring in data from a postgres instance into BigQuery. We are intermittently experiencing an error:
*** Log file is not found:…
sacoder
- 159
- 13
0
votes
0 answers
Migrating On Premise Apache Airflow workflow scripts to Cloud Composer
I have an on-premise environment running Airflow v2.2.0 and wish to migrate all the workflows in this instance to a Cloud Composer instance. While doing this migration, some of the operators used on the on-premise environment do not work after…
user538578964
- 723
- 9
- 25
0
votes
1 answer
Dynamic task creation using decorators in Airflow 2
I would like to create tasks dynamically in a flow based on the result from another task. Is it somehow possible to do it in the following way ?
I am able to do that by storing data from task in a temporary file and in the next task read it - but I…
Ania
- 139
- 1
- 9
0
votes
1 answer
Airflow - TaskGroup - looping through list of dictionaries to get multiple values - unhashable type: 'dict'
I'm trying to get the appropriate values in this list of dictionaries, which includes calling classes from 'table_name'. But I keep getting multiple errors, currently I'm seeing:
copy_VE_to_s3 = {group: S3CopyObjectsOperator( TypeError:
unhashable…
KristiLuna
- 1,601
- 2
- 18
- 52
0
votes
0 answers
Concurrent Airflow DAGS
Could anyone please let me know whether I can spin up many (say, around 10 concurrent DAGs) concurrent DAGs simultaneously in parallel at the same time? The parameters for these DAGs will be the same but with different values. Any ideas or…
testbg testbg
- 193
- 2
- 11
0
votes
0 answers
Broken DAG: [/usr/local/airflow/dags/my_dag.py] No module named 'airflow.operators.subdag'
I'm running airflow inside docker container and getting airflow image (puckel/docker-airflow:latest) from docker hub. I can access Airflow UI through localhost:8080 but without executing the DAG and the error mentioned in the subject above. I'm even…
Saurabh
- 199
- 1
- 3
- 11
0
votes
1 answer
Airflow 2 - MySQL connection issue
We have a MySQL task that is connecting to a 5.7.22 MySQL DB just fine in Airflow 1.10.15 using SSL (Extra: {"ssl": {"ca": "ca.pem"}} ). However, once we upgraded to Airflow 2.0.2 the task fails with (2026, 'SSL connection error:…
A Kong
- 3
- 1
- 3
0
votes
1 answer
remote logging doesn't work after upgrading Airflow version
We are using Airflow v2.1.4 with Kubernetes executors.
Our k8s cluster pods stdout automatically shipped via filebeat -> logstash -> Elasticsearch (ELK).
In logstash we are creating the log_id field:
mutate {
copy => { "[log][offset]" =>…
ItayB
- 10,377
- 9
- 50
- 77
0
votes
2 answers
Amazon MWAA and s3 connection
I'm migrating from on premises airflow to amazon MWAA 2.0.2. Currently I'm using an s3 connection which contains the access key id and secret key for s3 operations:
{
"conn_id" = "s3_default"
"conn_type" : "S3"
"extra" = {
…
Nisman
- 1,271
- 2
- 26
- 56
0
votes
2 answers
fresh install of Apache Airflow 2.2.3 .. OOps something bad happened
I have successfully gotten apache airflow installed locally via pip .. with a few needed pins
pip3 install zipp==3.1.0
pip3 install sqlalchemy==1.3.24
python3 -m pip install virtualenv
pip3 install apache-airflow[cncf.kubernetes]
pip3 install…
Erik
- 2,782
- 3
- 34
- 64
0
votes
1 answer
How to set mode=reschedule in SubdagOperator to get rid of deadlock
I see in the Airflow 2 SubdagOperator documentation link that using mode = reschedule we can get rid of potential deadlock.
To my understanding it is not a param which can be be passed with list of other params. If anyone has used this let me know…
codninja0908
- 497
- 8
- 29
0
votes
1 answer
Airflow DAG created using Taskflow failing with error: wrapper() missing 1 required positional argument: 'f'
I am trying to create a simple POC Airflow (2.2.3) DAG using Taskflow. Given below is the code:
from datetime import datetime
from airflow.decorators import dag, task
import yaml
import sys
import logging …
pallupz
- 793
- 3
- 9
- 25