Airflow user issues

Question

We have installed airflow from service account say 'ABC' using sudo root in virtual environment, but we are facing few issues.

• Calling python script using bash operator. Python script uses some environmental variables from unix account 'ABC'.While running from airflow, environmental variables are not picked. In order to find the user of airflow, created dummy dag with bashoperator command 'whoami', it returns the ABC user. So airflow is using the same 'ABC' user. Then why environmental variables are not picked?

• Then tried sudo -u ABC python script. Environmental variables are not picked, due to sudo usage. Did the workaround without environmental variables and it ran well in development environment without issues. But while moving to different environment, got the below error and we don't have permission to edit sudoers file. Admin policy didn't comply.

sudo: sorry, you must have a tty to run sudo

• Then used 'impersonation=ABC' option in .cfg file and ran the airflow without sudo. This time, bash command fails for environmental variables and it's asking all the packages used in script in virtual environment.

My Questions:

1. Airflow is installed through ABC after sudoing root. Why ABC was not treated while running the script.
2. Why ABC environmental variables are not picked?
3. Even Impersonation option is not picking the environmental variables?
4. Can airflow be installed without virtual environment?
5. Which is the best approach to install airflow? Using separate user and sudoing root? We are using dedicated user for running python script.Experts kindly clarify.

Answer 1

It's always a good idea to use virtualenv for installing any python packages. So, you should always prefer installing airflow in a virtaulenv.

You can use systemd or supervisord and create programs for airflow webserver and scheduler. Example configuration for supervisor:

[program:airflow-webserver]
command=sh /home/airflow/scripts/start-airflow-webserver.sh
directory=/home/airflow
autostart=true
autorestart=true
startretries=3
stderr_logfile=/home/airflow/supervisor/logs/airflow-webserver.err.log
stdout_logfile=/home/airflow/supervisor/logs/airflow-webserver.log
user=airflow
environment=AIRFLOW_HOME='/home/airflow/'

[program:airflow-scheduler]
command=sh /home/airflow/scripts/start-airflow-scheduler.sh
directory=/home/airflow
autostart=true
autorestart=true
startretries=3
stderr_logfile=/home/airflow/supervisor/logs/airflow-scheduler.err.log
stdout_logfile=/home/airflow/supervisor/logs/airflow-scheduler.log
user=airflow
environment=AIRFLOW_HOME='/home/airflow/'

Answer 2

We got the same issue as.

sudo: sorry, you must have a tty to run sudo

The solution we got is,

su ABC python script