Stack Overflow
Stack Overflow

Questions tagged [airflow-2.x]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.

690 questions
0
votes
0 answers

Migrations failed: When trying to install apt and pypi packages (both) in airflow

My dockerfile - FROM apache/airflow:2.2.3 COPY /airflow/requirements.txt /requirements.txt RUN pip install --no-cache-dir -r /requirements.txt USER root # Update aptitude with new repo RUN apt-get update # Install software RUN apt-get install…
Aditya Sinha
  • 614
  • 8
  • 18
0
votes
1 answer

Airflow:Run tasks in parallel dynamically based on number of dates

Requirement: Run tasks in parallel dynamically based on the number of offset values which is basically dates As below it starts from the current date 0 to 4 days back(end_offset_days), so that each task can run in parallel with each date in…
Karthik
  • 441
  • 5
  • 17
0
votes
2 answers

Airflow pool - less priority task is triggering first

I've been using the airflow pool to control my concurrent tasks. so I've created a test_pool with 10 slots and have created 4 tasks, out of which I have assigned 2 tasks with more priority weight by filling all the 10 slots each. However, the tasks…
Tula
  • 307
  • 6
  • 15
0
votes
2 answers

Airflow 2: Check if a task still exists or not with task_id

we have created a task for sensor operation, but the task name will be dynamically updated. i.e., f"{table_name}_s3_exists". We have a scenario where we have to check a table's location twice, but if the task is still present, we don't have to…
Santhosh
  • 71
  • 1
  • 5
0
votes
0 answers

Can't access the XCom on airflow "Error code: Out of Memory"

Well, for sure after trying my script sometimes, I forgot that the Airflow has a limit to load data and now I can't access the XComs to delete my data, is there a way to do that using the Ubuntu because I'm running my Airflow on LocalExecutor? Here…
Tayzer Damasceno
  • 302
  • 1
  • 4
  • 12
0
votes
2 answers

Airflow 2.2.3 - latency between tasks changes, tasks stay in queue too long [Cloud Composer]

We are using Cloud Composer in GCP (managed Airflow on a Kubernetes cluster) for scheduling our ETL pipelines. Our DAGs (200-300) are dynamic, meaning all of them are generated by a single generator DAG. In Airflow 1.x it was an antipattern due to…
Robert
  • 127
  • 2
  • 11
0
votes
1 answer

Airflow PythonOperator template_dict raises error TemplateNotFound(template)

I'm trying to pass bar.sql through the PythonOperator's template_dict for use in the python_callable, like the docs mention, but this is the closest example I've found. I've also reviewed this question which references Airflow 1.8, but the solution…
newtothis
  • 35
  • 10
0
votes
2 answers

How to get reason for failure using slack in airflow2.0

How to get the reason for the failure of an operator, without going into logs. As I want to post the reason as a notification through slack? Thanks, Xi
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
1 answer

How to get exact reason for failure in EMR through slack?

I need to have a slack notification when my pyspark code/ bootstrap fails with the exact reason for failure in EMR. I am using airflow 2.0, and I am using on_failure_callback=task_fail_slack_alert to notify slack. But it is not notifying. Is my…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
1 answer

On_callback_failure is not working within TaskGroup in airflow

How to implement on_callback_failure or similar parameter in TaskGroup in airflow? i want to implement on taskGroup level and not on DAG level.
Raul
  • 25
  • 5
0
votes
1 answer

How to check if the DAG is complete within Given time or not?

I have a Dag A, It runs at a time let's say 10 Am, and typically completes within 15-20 mins, but sometimes it takes more time and due to some tables in the Database it goes into an endless running state, how can I know that if my DAG is completed…
Aditya Verma
  • 201
  • 4
  • 14
0
votes
1 answer

Unable to see option to add a user in Airflow

I have installed MWAA 2.0 and I am the root user of this setup and I am not finding the option to add a user. I want to add a IAM user and so that he can trigger EMR from Airflow. How to add a existing IAM user to AIrflow (presuming that user has…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
2 answers

How to run selected parallel task at a time and to skip few task

I have a DAG which has 5 parallel task. I don't want to execute them all in one go. I want to pass some flag/value to the task suppose if the flag is set to True it runs and if the flag is set to False it gets skip. Can we do this in Airflow2. The…
Jaishree Rout
  • 382
  • 5
  • 17
0
votes
1 answer

Restrict the type of Tasks a User Role can run in Airflow

I am currently looking into airflow to allow a set of users to define & run some basic processes. However I would like to restrict the types of Tasks that those users are allowed to run. For example I do not want them to be able to use the…
user3188040
  • 671
  • 9
  • 24
0
votes
1 answer

DAG import error because ModuleNotFoundError

I recently upgraded from Airflow 1.10.15 to 2.2.4. Installed using sudo. Upgraded the DB after the upgrade. Airflow is running but there are import errors. I have multiple DAGs with the line from airflow.operators.mssql_operator import MsSqlOperator…
Parth Shah
  • 1,237
  • 10
  • 24
1 2 3
…
45 46