16

I have an EC2 instance that is running airflow 1.8.0 using LocalExecutor. Per the docs I would have expected that one of the following two commands would have raised the scheduler in daemon mode:

airflow scheduler --daemon --num_runs=20

or

airflow scheduler --daemon=True --num_runs=5

But that isn't the case. The first command seems like it's going to work, but it just returns the following output before returning to terminal without producing any background task: 

[2017-09-28 18:15:02,794] {__init__.py:57} INFO - Using executor LocalExecutor
[2017-09-28 18:15:03,064] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt
[2017-09-28 18:15:03,203] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt

The second command produces the error: 

airflow scheduler: error: argument -D/--daemon: ignored explicit argument 'True'

Which is odd, because according to the docs --daemon=True should be a valid argument for the airflow scheduler call.

Digging a little deeper took me to this StackOverflow post, where one of the responses recommends an implementation of systemd for handling the airflow scheduler as a background process according to the code available as this repo.

My lightly-edited adaptations of the script are posted as the following Gists. I am using a vanilla m4.xlarge EC2 instance with Ubuntu 16.04.3:

From there I call: 

sudo systemctl enable airflow-scheduler
sudo systemctl start airflow-scheduler

And nothing happens. While I have much more complex DAGs running on this instance, I am using this dummy case to create a simple test that also serves as a listener to let me know when the scheduler is operating as planned.

I've been using journalctl -f to debug. Here are a few lines of output from the scheduler process. There's no obvious problem, but my tasks aren't executing and no logs are being produced for the test DAG that would help me zoom in on the error. Is the problem in here somewhere? 

Sep 28 18:39:30 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:30,965] {dag_processing.py:627} INFO - Started a process (PID: 21822) to generate tasks for /home/ubuntu/airflow/dags/scheduler_test_dag.py - logging into /home/ubuntu/airflow/logs/scheduler/2017-09-28/scheduler_test_dag.py.log
Sep 28 18:39:31 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:31,016] {jobs.py:1002} INFO - No tasks to send to the executor
Sep 28 18:39:31 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:31,020] {jobs.py:1440} INFO - Heartbeating the executor
Sep 28 18:39:32 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:32,022] {jobs.py:1404} INFO - Heartbeating the process manager
Sep 28 18:39:32 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:32,023] {jobs.py:1440} INFO - Heartbeating the executor
Sep 28 18:39:33 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:33,024] {jobs.py:1404} INFO - Heartbeating the process manager
Sep 28 18:39:33 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:33,025] {dag_processing.py:559} INFO - Processor for /home/ubuntu/airflow/dags/capone_dash_dag.py finished
Sep 28 18:39:33 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:33,026] {dag_processing.py:559} INFO - Processor for /home/ubuntu/airflow/dags/scheduler_test_dag.py finished

When I run airflow scheduler manually this all works fine. Since my test DAG has a start date of September 9 it just keep backfilling every minute since then, producing a running time ticker. When I use systemd to run the scheduler as a deamon, however, it's totally quiet with no obvious source of the error.

Any thoughts?

aaron
  • 6,339
  • 12
  • 54
  • 80

2 Answers2

42

Documentation might be dated?

I normally start Airflow as following

airflow kerberos -D
airflow scheduler -D
airflow webserver -D

Here's airflow webeserver --help output (from version 1.8):

-D, --daemon Daemonize instead of running in the foreground

Notice there is not boolean flag possible there. Documentation has to be fixed.

Quick note in case airflow scheduler -D fails:

This is included in the comments, but it seems like it's worth mentioning here. When you run your airflow scheduler it will create the file $AIRFLOW_HOME/airflow-scheduler.pid. If you try to re-run the airflow scheduler daemon process this will almost certainly produce the file $AIRFLOW_HOME/airflow-scheduler.err which will tell you that lockfile.AlreadyLocked: /home/ubuntu/airflow/airflow-scheduler.pid is already locked. If your scheduler daemon is indeed out of commission and you find yourself needing to restart is execute the following commands: 

sudo rm $AIRFLOW_HOME airflow-scheduler.err  airflow-scheduler.pid
airflow scheduler -D

This got my scheduler back on track.

aaron
  • 6,339
  • 12
  • 54
  • 80
Tagar
  • 13,911
  • 6
  • 95
  • 110
  • 2
    Good suggestion, but this doesn't fix it. None of the processes are running as they do when I run `airflow scheduler` manually without the daemonize flag. What else could this be? `which airflow` produces this: `/home/ubuntu/.local/bin/airflow` – aaron Sep 29 '17 at 00:51
  • I'd be interested to know what you're experience is here. When I run `airflow scheduler -D` the airflow pretty print pops up, but then instead of the process running I'm returned to the terminal prompt. I'm not given in PID indicating where the scheduler is running, nor are any of my processes executing. What happens for you? – aaron Oct 04 '17 at 22:13
  • If it's not starting for you, check $AIRFLOW_HOME/airflow-scheduler.err - it should contain error why it didn't start (it might be because pid file already exists, or it can't connect to backend database or something else). Yes, it prints airflow banner and then starts in background. If you still can't get it running, try using `strace -f airflow scheduler -D` to see if you can spot why it's failing. – Tagar Oct 04 '17 at 22:20
  • Working great, thank you. It's been on for a couple days without a hitch :) – aaron Oct 07 '17 at 07:29
  • @Tagar - where do the logs go? I tried your method and it works fine, but can't see my worker logs. I'm using a multi-node Celery executor. It works fine when I merely call each of the processes that need to run. However, daemonizing them was a pain and this seems like a good inbetween solution. I even did `airflow worker -D >> /path/to/log/file.log &` and while the the file is there, it doesn't show the normal depth of information I usually get. Please help! – CodingInCircles Apr 04 '18 at 19:59
  • I also had to remove airflow-webserver-monitor.pid – routeburn May 13 '19 at 18:40
  • thanks for your suggestion. I am running Bitnami Airflow multi-tier on Azure. The VM is a Linux box. I was suffering from somewhat the same issue. In my case all I did was remove `sudo rm $AIRFLOW_HOME airflow-scheduler.err airflow-scheduler.pid` and then run `airflow scheduler -D` – Jiraheta Jul 16 '22 at 04:02
3

About task start via systemd:

I had a problem with the PATH variable when run this way is initially empty. That is, when you write to the file /etc/sysconfig/airflow:

PATH=/home/ubuntu/bin:/home/ubuntu/.local/bin:$PATH

you literally write:

PATH=/home/ubuntu/bin:/home/ubuntu/.local/bin

Thus, the variable PATH doesn't contain /bin which is a bash utility that LocalExecutor uses to run tasks.

So I do not understand why in this file you have not specified AIRFLOW_HOME. That is, the directory in which the Airflow is looking for its configuration file.

MSemochkin
  • 165
  • 8
  • I haven't had the chance to try this yet but will get back to you ASAP as soon as I'm able to give it a shot. If it fixes the problem I'll definitely pick your answer. – aaron Oct 03 '17 at 15:39
  • Tagar's answer was a simpler way to resolve this. Thank you for your suggestion, though! – aaron Oct 07 '17 at 07:30