Stack Overflow
Stack Overflow

Questions tagged [airflow-2.x]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.

690 questions
2
votes
0 answers

Can't access to XCOM view in Airflow 2.1.2

When I go to Admin --> Xcom section in Airflow 2.1.2 Webserver UI I get this error: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app response =…
mrc
  • 2,845
  • 8
  • 39
  • 73
2
votes
0 answers

How to pass a conf to a scheduled DAG

When the DAG is triggered manually there are multiple ways to pass the config. It could be done from the UI, via the airflow CLI using --conf argument & using the REST API. But when a DAG is scheduled using a cron expression, the DAG always fails…
user3583252
  • 493
  • 1
  • 5
  • 15
2
votes
2 answers

Triggering Airflow DAG via API

I have installed Airflow 2.0.1 on EC2 with PostgreSQL RDS as metadata db. I want to trigger DAG from Lambda so tried to test the code with curl but am receiving Unauthorized as response. What if anything should I be doing…
Ronak Shah
  • 51
  • 1
  • 4
2
votes
1 answer

Airflow 2.2.0 Install Error: package versions have conflicting dependencies cattrs

I am getting below error while trying to follow the installation steps at https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html ERROR: Cannot install apache-airflow[celery]==2.2.0 because these package versions…
Krishna Pandey
  • 360
  • 1
  • 6
  • 19
2
votes
2 answers

Airflow with KubernetesExecutor: Recorded pid does not match the current pid.airflow.Getting AirflowException: Task received SIGTERM signal

I am running Airflow:2.1.4 on a KubernetesExecutor with Kubernetes: 1.21.0 Getting AirflowException: Task received SIGTERM signal Stack trace: [2021-10-11 06:22:52,543] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 17 [2021-10-11…
Rajeev Reddy
  • 21
  • 2
2
votes
1 answer

Airflow 2 Push Xcom with Key Name

In Airflow 2 taskflow API I can, using the following code examples, easily push and pull XCom values between tasks:- @task(task_id="task_one") def get_height() -> int: response = requests.get("https://swapi.dev/api/people/4") …
Indrid
  • 962
  • 4
  • 25
  • 39
2
votes
2 answers

Delay between tasks in Airflow or any other option?

We are using airflow 2.00. I am trying to implement a DAG that does two things: Trigger Reports via API Download reports from source to destination. There needs to at least 2-3 hours gap between tasks 1 and 2. From my research I two options Two…
user2452057
  • 816
  • 2
  • 11
  • 23
2
votes
1 answer

Airflow 2.0.1 : Write logs in Local Timezone

By default, airflow 2.0.1 write logs in UTC timezone. But I want the logs to write in Local/Machine [non UTC] timezone. I tried changing below. But no luck AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE: Asia/Kolkata AIRFLOW__CORE__DEFAULT_TIMEZONE:…
Ganesh
  • 677
  • 8
  • 11
2
votes
2 answers

Create dynamic workflows in Airflow with XCOM value

Now, I create multiple tasks using a variable like this and it works fine. with DAG(....) as dag: body = Variable.get("config_table", deserialize_json=True) for i in range(len(body.keys())): simple_task = Operator( …
Nontaphat S.
  • 45
  • 1
  • 6
2
votes
1 answer

How to disable default authentication on airflow 2.0.1

We are upgrading our airflow service to 2.0.1 deployed on AWS, However, after upgrading, it has default authentication page. airflow2 We have already set authenticate and rbac to false in the configure file for our ansible deployment as suggested by…
binbin2046
  • 21
  • 2
1
vote
1 answer

How to MagicMock Airflow context in python?

I am a new-bee to Python/Airflow and trying to use MagicMock for unit test cases. I was able to use it in few cases but in one case I am struggling to get it working. Method which I want to test (UT): def resolve_sensitive_params(self,…
MHegde
  • 329
  • 3
  • 14
1
vote
1 answer

Apache airflow issue while running airflow helm chart 8.7.0 and web server in not running it is stuck on worker exit and restarting

`I am using airflow 8.7.0 helm chart and up airflow pods but the web server pod is restarting after giving worker exit error and Received signal: 15. Closing gunicorn. Error Logs [2023-07-10 05:42:16,665] {providers_manager.py:235} INFO - Optional…
Indrajeet singh thakur
  • 11
  • 2
1
vote
0 answers

Cannot find DAG on airflow dags list

enter image description here It's the first time I tried to create airflow dag but when I checked the dag lists in terminal, there is no dag_id that I created Is there anything wrong with my file or the directory of the dag the I created
Rattapong Pojpatinya
  • 11
  • 1
1
vote
1 answer

Airflow DAG does not start

I have created a couple of DAGs, almost identical, just to test scheduling. Created the DAG files on 21 June 2023, at 1:22 and 1:57 AM (night, local time in Ukraine). Their start times: start_date = datetime(2023, 6, 21, 10, 0, 0) and start_date =…
user3791111
  • 1,469
  • 1
  • 15
  • 20
1
vote
1 answer

How to get the initial start-time and end-time for Airflow DAG, including all tasks

How can I get the start and end time of the DAG in overall, which includes all the tasks (which is the initial task start time and end time of the last task)?
Austin Jackson
  • 153
  • 6
1 2 3
…
45 46