2

I'm trying to run this upstart script from the official repo for airflow: https://github.com/apache/incubator-airflow/blob/master/scripts/upstart/airflow-webserver.conf

start on started networking
stop on (deconfiguring-networking or runlevel [016])

respawn
respawn limit 5 30

setuid airflow
setgid airflow

exec usr/local/bin/airflow webserver

When I run it this is the output:

$ sudo service airflow-webserver start
airflow-webserver start/running, process 7612

$ sudo service airflow-webserver status
airflow-webserver stop/waiting

But just doing exec /usr/local/bin/airflow webserver will run it perfectly without any hiccups, which is puzzling. Does anyone know why this is happening?

* You can get airflow with pip install airflow to test in case you need to see for yourselves.

m0meni
  • 16,006
  • 16
  • 82
  • 141
  • Does airflow still work as intended despite the message? – J. Allan Jun 27 '16 at 20:56
  • @JefréN. unfortunately not. – m0meni Jun 27 '16 at 20:58
  • In that top 'codeblock', does it really say `exec usr/local/bin/airflow webserver` instead of `exec /usr/local/bin/airflow webserver`? – J. Allan Jun 27 '16 at 21:02
  • @JefréN. yes, but it actually doesn't matter for the upstart script. I've figured out the problem, which is that I neglected that upstart does not get environment variables from `/etc/profile`, and I needed to set them in the file. – m0meni Jun 27 '16 at 21:03

2 Answers2

2

Upstart runs in a clean environment, meaning that it won't use the variables in /etc/environment. This means you have to set AIRFLOW_HOME and AIRFLOW_CONFIG again.

description "Airflow webserver daemon"

start on started networking
stop on (deconfiguring-networking or runlevel [016])

respawn
respawn limit 5 30

setuid airflow
setgid airflow

# I omitted all of the below because I assumed 
# it would pick it up from the already defined env
env AIRFLOW_CONFIG=/path/to/airflow/airflow.cfg
env AIRFLOW_HOME=/path/to/airflow
export AIRFLOW_CONFIG
export AIRFLOW_HOME

exec usr/local/bin/airflow webserver
m0meni
  • 16,006
  • 16
  • 82
  • 141
0

Here is what I did in Ubuntu 16: (The init scripts are taken from the incubator-airflow git repo)

$ sudo cp incubator-airflow/scripts/systemd/*.service /etc/systemd/system/

In each service file, change user and group lines to the owner of directory, in my case - ubuntu and modify ExecStart:

User=ubuntu

Group=ubuntu

ExecStart=/usr/local/bin/airflow  webserver --pid $AIRFLOW_HOME/webserver.pid

keep in mind that ExecStart line is different in every service line.

Then:

$ sudo cp incubator-airflow/scripts/systemd/*.conf /etc/tmpfiles.d/
$ sudo mkdir /etc/sysconfig
$ sudo cp incubator-airflow/scripts/systemd/airflow /etc/sysconfig/airflow

Add the following lines to /etc/sysconfig/airflow: 

AIRFLOW_CONFIG=YOUR_airflow_home/airflow.cfg

AIRFLOW_HOME=YOUR_airflow_home

Finally:

sudo systemctl daemon-reload
sudo systemctl start airflow-webserver

(Or any other airflow service)

Good Luck!

Natalie Penso
  • 41
  • 2