Stack Overflow
Stack Overflow

Questions tagged [airflow-webserver]

161 questions
0
votes
0 answers

Want to go ahead with Airflow Production, need guidance on parameters

Please, we need input and guidance: We needed input for the Airflow production setup, as we were able to accomplish a small POC on SQLite and a sequential executor as well as local executor in AirFlow 2.5.0. Well, we have almost 10 concurrent jobs,…
Austin Jackson
  • 153
  • 6
0
votes
0 answers

Airflow is having deadlock in backend database with MSSQL , Same Dag is working fine in postgresql backed

`I am trying to run a Dag with MSSQL version 19, backend and airflow version as 2.5.0 below is the DAG: ``*import pendulum from airflow import DAG from airflow.operators.python_operator import PythonOperator from datetime import timedelta # Define…
rahul ranjan
  • 1
  • 1
0
votes
0 answers

scheduler in Airflow apach

Running multiple dags with different id set to run every minute. Such statistics first release 23:58:26 next run 23:59:26 and then it started twice (according to the schedule logs) but this launch is not displayed on the web and then he reset to a…
Andrey
  • 1
0
votes
1 answer

airflow interface DAGS keep failing

I am new to airflow and am trying to get everything set up. I am currently working through a tutorial that gives a basic overview of DAGS and airflow functionality. I have airflow set up on a linux server and have my tutorial DAG populating in the…
Curtis Wiser
  • 63
  • 2
0
votes
0 answers

Airflow no module found error 'airflow.providers.airbyte' on UI

Broken DAG: [/usr/local/airflow/dags/dag_airbyte_example.py] Traceback (most recent call last): File "", line 241, in _call_with_frames_removed File "/usr/local/airflow/dags/dag_airbyte_example.py", line 3, in…
Shruti Kulkarni
  • 11
  • 3
0
votes
0 answers

Airflow logs not visbile in UI while hosting in AWS

botocore.parsers.ResponseParserError: Unable to parse response (not well-formed (invalid token): line 1, column 0), invalid XML received. Airflow log configurations are set to remote s3. Log files are saved in s3 but it is loading in Airflow UI
Matrix
  • 9
  • 1
  • 4
0
votes
1 answer

Passing Parameter from conf in Airflow PostgresOperator

Want to access the config passed in the data in the API call curl --location 'http://localhost:8080/api/v1/dags/postgres_operator_dag/dagRuns' \ --header 'Content-Type: application/json' \ --data '{ "conf": { "seller_id": "test_seller" …
Arjunsingh
  • 703
  • 9
  • 22
0
votes
0 answers

Can Not change Airflow Log Level in KubernetesPodOperator by using docker-compose env variables

I am using docker-compose to deploy a local Airflow instance by following the official documentation. The default Airflow logging level is WARNING. Hence in docker-compose I added the following environment variable to set the log level to…
Hui
  • 97
  • 7
0
votes
0 answers

How to create an Airflow AWS connection using .aws credentials in the home directory on Airflow Web UI?

I am running a local Airflow test environment. I copied my aws credentials to the Docker container that runs the Airflow scheduler, the web server and the worker respectively under the following directories: /home/airflow/.aws /home/.aws How do I…
Hui
  • 97
  • 7
0
votes
0 answers

How to set up Airflow Connections aws_default & emr_default to run spark job create & terminate EMR cluster ? Why we need two connection?

I am trying run Spark Job from Airflow on EMR cluster using EMR steps , what configuration you should put in aws_default & emr_default .... Access key & Secure key ? Only ? Or ports ? Extra ? Why it needs these two separate connections ? How to…
dol
  • 21
  • 2
0
votes
1 answer

airflow airbyte operator - module not found

Context I'm using Airflow hosted on Google kubernetes engine, I've added apache-airflow-providers-airbyte 3.2.0 to the requirements.txt which is installed and appears on the Airflow - Admin > Providers" UI along with the default…
IOIOIOIOIOIOI
  • 277
  • 3
  • 15
0
votes
1 answer

Airflow: Outdated DAGs persisting in Web UI after gitSync

I am using Airflow on Kubernetes using git-sync to sync DAGs from a git repository. I can successfully import DAGs, but I'm seeing an issue where old changes are persisting in the Airflow UI alongside new changes. Example: In my remote repository I…
user1763430
  • 77
  • 1
  • 7
0
votes
1 answer

error when trying to modify airflow.cfg in composer - Composer Backend timed out. Multiple errors occurred

I'm trying to configure sending emails through smtp, but I need to add a series of parameters in airflow.cfg, but whenever I make the insertions in the modifications tab of the airflow configuration, I always get the error below, I already have full…
Luiz Felippe
  • 1
  • 3
0
votes
0 answers

Airflow task stuck in queued state and not running

I see airflow task getting stuck in queued state and not running. We are running Version : 1.10.15. It was all running fine until 2 days ago and we started seeing this issue. What could be causing this issue ?
Imran
  • 121
  • 4
  • 9
0
votes
0 answers

Airflow on docker container, problem with the http protocol

I try to run a dag each 2 minutes, but I have an error on the http protocol when a task tries to run : Failed to fetch log file from worker. Request URL is missing an 'http://' or 'https://' protocol The strange thing is that if I put explicitly…
zolora
  • 33
  • 2
1 2 3
…
10 11