Stack Overflow
Stack Overflow

Questions tagged [airflow-2.x]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.

690 questions
-1
votes
0 answers

Unit testig Airflow configuration

Task: Write unit tests importing DAGs and checking their validity in the CI pipeline, similar to https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#unit-tests Problem: My tests work locally but fail in the pipeline (failing to…
matwasilewski
  • 384
  • 2
  • 11
-1
votes
1 answer

Error while connecting to Oracle DB when running DAG in Airflow 2.6 which was started as systemd/systemctl service

Okay, so this question is multi-parted due to the nature of the situation, so please please be patient and I will try to clarify as many details as I can through the question and comments. If the question needs to be in another Stack-forum, please…
Samuel Gottipalli
  • 51
  • 7
-1
votes
1 answer

Airflow: restrict viewing of user dags

im using Airflow:2.5.3 I have assigned subfolders for each user inside the dags folder of airflow dags/ ├── dag-user1/ │ ├── dag_a.py | ├── dag_b.py ├── dag-user2/ | ├── dag_c.py | ├── dag_d.py ├── dag-user3/ | ├── dag_e.py | ├──…
Krengifo
  • 11
  • 2
-1
votes
1 answer

intentionally raise an exception and fail the DAG

In my Airflow dag, I am doing this: try: response = requests.get(url, params=params, headers=headers) if response.status_code == 403: raise…
x89
  • 2,798
  • 5
  • 46
  • 110
-1
votes
2 answers

Can we upload file through airflow web-server UI and trigger a DAG?

I need to upload a log file through the airflow web server UI and parse that log file in a DAG. My requirement is a button on UI, which upon clicking will open a file selector to upload a file to airflow. then use this file in the DAG. Similarly, we…
Prinz Piuz
  • 69
  • 9
-1
votes
2 answers

airflow - Dags are not being executed on scheduled time

I wanted to have my dags to have first run on 2:00 AM on 25th and then onwards Tuesday to Sat daily run at 2:00 am. following is how my scheduling look like. with DAG( dag_id='in__xxx__agnt_brk_com', schedule_interval='0 2 * * 2-6', …
Gaurang Shah
  • 11,764
  • 9
  • 74
  • 137
-1
votes
1 answer

How to print Airflow time?

Need this info in the log as a print statement click for more info
Roshan_mobo
  • 3
  • 2
-1
votes
1 answer

How to trigger airflow dynamic dag with configs passed from lambda

I have a dynamic dag using which I am creating multiple dags(different workflows) based on the value passed from lambda with their config. How can I trigger the dynamic dag script and trigger the dag from lambda
anukriti kulshrestha
  • 29
  • 3
-1
votes
1 answer

Installing local .whl files on Databricks cluster

I am trying to connect to a databricks cluster and install a local python whl using DatabricksSubmitRunOperator on Airflow (v2.3.2) with following configuration. However, it doesn't work and throws a fileNotFound exception (I checked file path…
Dhrumil
  • 67
  • 11
-1
votes
1 answer

GCP Storage - unable to read contents of file from Google storage bucket

I've a requirement wherein i have to call a python file stored in GCP Storage bucket, from the driver python file. Here is the code(being run on my local m/c, but eventually will be scheduled using Airflow on GCP): from google.cloud import…
Karan Alang
  • 869
  • 2
  • 10
  • 35
-1
votes
1 answer

Composer Airflow - Cross DAG Task Dependancy

I have below 2 DAGs and tasks DAGA - Task1,Task2,Task3 DAGB - Task4,Task5,Task6 Now Task4 & 5 of DAG B depends on Task1 of DAGA and Task6 depends on Task3 but I want this dependency only on Monday and for remaining days I don't want this dependency.
Rahul Wagh
  • 281
  • 6
  • 20
-1
votes
1 answer

airflow task is failing because of "Set is not JSON serializable"

I got a DAG which is using BigQuery operators in some of its tasks. One of such tasks keeps failing, saying that my configuration parameter is actually a set, which cannot be serialised to JSON. However, I don't understand why I'm getting the error,…
Dasph
  • 420
  • 2
  • 15
-1
votes
1 answer

Cannot import name 'get_default_executor' from 'airflow.executors'

In Airflow 1.10.15 using celery executor in docker, the "apache-airflow-upgrade-check" package was used for the checks required to upgrade to Airflow 2.2.3. Below image is the output after executing the command. The Airflow Config file is updated as…
VAR
  • 1
-2
votes
1 answer

install airflow on compute engine

I would like to ask if any of you have experience installing Airflow on Compute Engine. I have been searching for instructions on Google and asking ChatGPT, but I have not been successful so far. I would like to know the correct sequence of commands…
linggapratama28
  • 23
  • 6
1 2 3
…
45
46