8

Airflow scheduler is picking up the dags from the correct folder as per set in the airflow.cfg file. However, Airflow ui webserver is picking the dags from wrong folder. I realised this via looking at the stdout of the terminal in which airflow webserver is running: [2018-10-08 14:26:48,519] [15535] {models.py:167} INFO - Filling up the DagBag from /home/aviral/airflow/dags

Note, my airflow.cfg file has:

# The folder where your airflow pipelines live, most likely a
# subfolder in a code repository
# This path must be absolute
dags_folder = /home/aviral/dev/airflow_learning/airflow_home/dags
Meghdeep Ray
  • 5,262
  • 4
  • 34
  • 58
aviral sanjay
  • 953
  • 2
  • 14
  • 31

2 Answers2

11

It looks to me like you have multiple Airflow installations. I think you have to check the UI in the Admin -> Configuration menu.

That will tell you which Airflow is active and set as AIRFLOW_HOME and AIRFLOW_CONFIG.

If the AIRFLOW_CONFIG environment variable is not set, it will always default to your home directory. You'll have to set it to something like this: AIRFLOW_CONFIG=$AIRFLOW_HOME/airflow.cfg.

Meghdeep Ray
  • 5,262
  • 4
  • 34
  • 58
4

Change the Dag folder by using the following commands.

Open the .bashrc or .zshrc user file of the terminal

vi ~/.bashrc

Add the below commands in the file

 export AIRFLOW_HOME='/var/www/html/airflow'
 export AIRFLOW_CONFIG=$AIRFLOW_HOME/airflow.cfg

Run the .bashrc file by using the below command.

source ~/.bashrc

Now if you run airflow webserver, it will pick the dags from the AIRFLOW_HOME/dags directory.

Lijo Abraham
  • 841
  • 8
  • 30