Stack Overflow
Stack Overflow

Questions tagged [airflow-webserver]

161 questions
1
vote
0 answers

Unable to Delete Airflow Dags using Airflow Session

I want to write down airflow dags cleanup dag. I' https://github.com/teamclairvoyant/airflow-maintenance-dags for reference. I have written a cleanup DAG which takes all the dags and check their schedule, if it's once, it deletes the DAG. When I'm…
Akash Srivastava
  • 23
  • 2
1
vote
1 answer

How can I access a variable defined in a function[airflow Pythonoperator called function] and use it outside the airflow-template scope?

## Section 1 | Import Modules ## Section 2 | DAG Default Arguments ## Section 3 | Instantiate the DAG ## Section 4 | defining Utils ## Section 5 | Task defining ## Section 6 | Defining dependecies ## Section 1 | Import Modules from airflow…
Sourav Roy Chowdhury
  • 143
  • 1
  • 6
1
vote
0 answers

Apache Airflow - Any have a scrip with tables of metadata?

have a problema when i need create all tables of metadata on airflow. I created a docker-compose that points to on premise mariadb-mysql and when i up the compose, some tables don't create all variable or directly do not create the tables This is…
Joaquin
  • 49
  • 4
1
vote
1 answer

How to schedule execution without delay?

I have one problem, I need to launch one DAG every first day of month, but I have one problem, the DAG started on 1 October but executed that day on 1 November, I need that 1 October execute 1 October and 1 November execute 1 November, and not delay…
Guille
  • 23
  • 2
1
vote
2 answers

How to set up airflow worker to allow webserver fetch logs on different machine with docker?

I just recently installed airflow 2.1.4 with docker containers, I've successfully set up the postgres, redis, scheduler, 2x local workers, and flower on the same machine with docker-compose. Now I want to expand, and set up workers on other…
Zheng
  • 13
  • 1
  • 5
0
votes
0 answers

Python Tasks in MWAA Airflow are failing

I have a mwaa airflow environment that is running. I have setup the logs. All the logs seem to run in cloud watch except for dag tasks. When I create a simple pythonoperator dag script (a function that prints hello world) the tasks fails and does…
SRUSTY
  • 1
0
votes
0 answers

Setting airflow web_server_ssl_cert and web_server_ssl_cert key Values.yaml in Helm (Kubernetes)

I am running chart =>Helm Chart version 3.2.10 and Apache-airflow/airflow 2.6.2 on K8s , I am trying to set web_server_ssl_cert and web_server_ssl_key in the Values.yaml to map the airflow.cfg Created secret => kubectl create secret generic…
Samson Eromonsei
  • 1
  • 1
0
votes
0 answers

When adding an additional sub task to an already created dag in airflow, is there a way to run only the sub task at the previous time?

I want to add a new sub task to an already running dag and make it work. Since each task processes data based on a logical date, each task must be fully executed after the start date... But I want to run only the sub task of the already completed…
Jmob
  • 49
  • 7
0
votes
0 answers

I am unable to see SSH connection in connection list in Airflow

I have installed the SSH package in airflow with command pip install apache-airflow-providers-ssh. Also restarted web server and scheduler couple of times. I do see the list of SSH package installed with command pip package list but unable to see…
Swati Ojha
  • 1
0
votes
0 answers

is there alternate for sklearn import in intellij with airflow and docker

attached screenshot I'm facing ModuleNotFoundError: No module named 'sklearn' in intellij as I'm using docker and airlfow to create a machine learning project eventhough I;ve installed package in the intellij and in docker container as well.…
Ashwini Choudhary
  • 15
  • 2
0
votes
0 answers

Override/update one of the params through a file in Airflow

How to update the params on the fly? I want to update 'input_file' param to the path present in file 'temp_file.txt' import datetime as dt from airflow import DAG from airflow.providers.ssh.operators.ssh import SSHOperator from…
tanviz1992
  • 1
  • 1
0
votes
1 answer

How can I set up the default redirect URL in Airflow?

I am currently running Airflow within a docker compose, and it is accessible at http://localhost:8081. However, whenever I access this URL, I am automatically redirected to http://localhost:8081/login/?next=http%3A%2F%2Flocalhost%3A8081%2Fhome. This…
mkUltra
  • 2,828
  • 1
  • 22
  • 47
0
votes
1 answer

Airflow Dag isn't completing its execution. [heartbeat] error

I am trying to take baby steps with Airflow. I was able to test the task successfully via VS Terminal. However, when I trigger the Dag trough UI it just stay in 'running' stage forever. When I forcefully mark as failed and review logs. I see it as…
Sri_Dev
  • 1
  • 2
0
votes
0 answers

Task level failures are not triggered, in fact the dag level is repeated with alert TWICE

from datetime import timedelta from datetime import datetime from airflow import DAG from airflow.decorators import task, dag from airflow.providers.ssh.operators.ssh import SSHOperator from airflow.operators.bash_operator import BashOperator from…
Austin Jackson
  • 153
  • 6
0
votes
0 answers

How can I troubleshoot the issue of Airflow jobs getting unexpectedly terminated after migration?

let me explain Airflow upgradation and migration story here. I am using airflow version 1.10.3 with python 2.7 version to run my shell scripts, Apache Spark jobs etc. All jobs are working fine in current airflow version. I recently started upgrading…
Vijay_Shinde
  • 1,332
  • 2
  • 17
  • 38
1 2 3
…
10 11