Stack Overflow
Stack Overflow

Questions tagged [airflow-webserver]

161 questions
0
votes
1 answer

HTTPS configuration in Amazon Linux

I am new to devops. I have a scenario, where i need to install Airflow and open https to access globally. Below are the steps i done. Created EC2 and installed Airflow Install Httpd and opened 443 port and restarted httpd Generated ssl…
Charles
  • 9
  • 4
0
votes
0 answers

Airflow error Command exited with return code 127

I have a Ubuntu Server with airflow installed, I have a basic dag deployed there, the problem appear when I rebooted in the server and when I tried again to run the dag, airflow is showing this error: [2022-11-29, 00:01:54 UTC] {subprocess.py:75}…
Brahian
  • 46
  • 1
  • 5
0
votes
1 answer

Airflow - Failed to fetch log file from worker. 404 Client Error: NOT FOUND

I am not able to fetch the worker pod logs on airflow UI. The error I am getting on airflow is: ` *** Falling back to local log *** Trying to get logs (last 100 lines) from worker pod aggregationtestcheckingcache-33ea24f45f1344d7a628e21a53b4f6d0…
Bharat Goel
  • 1
0
votes
0 answers

Let webserver look at fresh pip install

I'm working on a poc to work with airflow on k8s. I'm missing an pip package and I'm adding that through the shell in kubectl. That works and when i look in the shell and do pip list i see the new package. Im adding it to the webserver of…
Eindbaas
  • 23
  • 4
0
votes
1 answer

Airflow fault tolerance

I have 2 questions: first, what does it mean that the Kubernetes executor is fault tolerance, in other words, what happens if one worker nodes gets down? Second question, is it possible that the whole Airflow server gets down? if yes, is there a…
Mohammed Abdulwahab
  • 1
0
votes
0 answers

Airflow Webserver cant fetch log from Worker Celery

I have a problem with Apache Airflow Logging. My Airflow cluster has one Webserver, one Scheduler and five Workers Celery. All Workers working usually but only four Workers can fetch the log, one worker cant fetch the record with error: "Failed to…
Nguyễn Hùng
  • 13
  • 3
0
votes
0 answers

Airflow 2.4.1 Helm - Dags from PVC not visible in Web UI

I'm trying to setup Airflow 2.4.1 on AKS using official Helm, I've managed to standup pods for scheduler, triggerer, external Postgres(using separate helm). After installing airflow helm, db migrations are run successfully as relevant tables are…
S N
  • 540
  • 1
  • 6
  • 18
0
votes
0 answers

Apache Airflow 2.1.2 dags are not refreshing after I restarted my vm where airflow is hosted

Airflow is not able to refresh the dags. Airflow is hosted on azure vm and filestorage is mounted and added those mounts in env file. Eearlier things were working file. as soon as I uplaod an updated script, changes would show up. But now it…
Manthan Gajjar
  • 1
  • 1
0
votes
1 answer

How to use airflow secret backend with aws secret manager

I have installed the airflow 1.10.15 in standalone server & trying to integrate aws secret manager with it but values are not coming. I have added backend = airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend and backend_kwargs =…
Saurabh Srivastava
  • 175
  • 8
0
votes
1 answer

Airflow XCOMs communication from BashOperator to PythonOperator

I'm new to Apache Airflow and trying to write my first Dag which has a task based on another task (using ti.xcom_pull) PS : I run Airflow in WSL Ubuntu 20.04 using VScode. I created a task 1 (task_id = "get_datetime") that runs the "date" bash…
Salim Mzoughi
  • 41
  • 4
0
votes
0 answers

HttpSensor if endpoint not accessible or conn is wrong/not defined then do not go into next task

I have following DAG below. My concern is about HttpSensor. My target is if the endpoint is not accessible for whatever reason i do not want want the next task to be reached get_data. Moreover i would like sensor to check enpoint continously and…
Henry
  • 537
  • 1
  • 9
  • 22
0
votes
1 answer

The conn_id envinorment variable isn't defined

I have the dag which tries connects to http endpoint nevertheless somehow it doesn't work. I defined connection using envinorment variable but somehow it is not seen by httpsensor and i am getting below error however that variable was created in the…
Henry
  • 537
  • 1
  • 9
  • 22
0
votes
2 answers

DAG Has Yet to Run - Airflow

I'm new to airflow, really appreciate any help for the following problem. I tried to run the airflow webserver on my laptop. Theoratically, I set the start_time=datetime.now(), it should be run successfully when I manually ran the dag on the…
lynn kuo
  • 11
  • 1
0
votes
0 answers

Airflow dags being deleted

I have airflow installed on kubernetes cluster. Ive created some dags nevertheless any time i restart server for example by: cat $AIRFLOW_HOME/airflow-webserver.pid | xargs kill -9 server deletes all my dags i created. How to keep dags even on…
Henry
  • 537
  • 1
  • 9
  • 22
0
votes
0 answers

Airflow - Druid Operator is not getting host

I have a question for Druid Operator. I see that this test is successful, but I take a this error. File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 792, in get_adapter raise InvalidSchema(f"No connection adapters…
Emin Can OĞUZ
  • 1
  • 2
1 2 3
…
10 11