How to stop/kill airflow scheduler started in daemon mode

Question

I and new to airflow and accidentally started airflow scheduler in daemon mode. Now, I want to kill the scheduler and possibly restart it. I tried doing

    sudo kill -9 <list of pids>
    pkill <name>

nothing is happening. When I run

    ps aux | grep 'airflow scheduler'

I see these entries:

    user1   2907  6.0  1.0 329788 62996 ?        Sl   17:37   1:26 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D
    user1   2909  0.0  0.9 327576 58948 ?        Sl   17:37   0:00 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D
    user1   2910  0.0  0.9 327576 58944 ?        Sl   17:37   0:00 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D
    user1   2911  0.0  0.9 327576 58944 ?        Sl   17:37   0:00 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D

...and so on for 35 lines with different pids.

Any recommendation as to how I can stop/kill airflow scheduler without restarting my machine. I have also checked the pid file for scheduler and tried killing that pid but no effects.

Any help is appreciated. Thanks!

Did you try killing everything with airflow using something like ps -ef | grep airflow | awk '{print $2}' | xargs kill -9 — Him, Jun 26 '17 at 09:05

score 15 · Answer 1 · answered Jan 18 '18 at 17:13

15

Unfortuntely

kill $(ps -ef | grep "airflow scheduler" | awk '{print $2}')

I was not able to find a clean solution.

Also looking into the code

https://github.com/apache/incubator-airflow/blob/master/airflow/bin/cli.py

answered Jan 18 '18 at 17:13

venergiac

7,469
2
48
70

score 6 · Answer 2 · edited Apr 26 '20 at 15:31

6

Go to the airflow directory where the pid file is and use: cat airflow-scheduler.pid | xargs kill

edited Apr 26 '20 at 15:31

portrounding

3
2

answered Oct 07 '19 at 23:10

spennybuns

197
2
7

Alejandro Kaspar · Answer 3 · 2021-02-27T01:20:02.670

Another alternative is:

/usr/bin/rm -f ${AIRFLOW_HOME}/airflow-scheduler.pid
/usr/bin/pkill -f "airflow-scheduler"

Please note better to remove older daemon .pid file; to avoid any issue while restarting on daemon mode.

You can also see: https://github.com/apache/airflow/issues/77

/bin/sh -c 'PATH=/bin:/sbin:/usr/bin:/usr/sbin  mPPID=`cat ${AIRFLOW_HOME}/airflow-scheduler.pid`;ps -o pid= --ppid $mPPID | xargs kill -15 && kill -15 $mPPID && rm -f ${AIRFLOW_HOME}/airflow-scheduler.pid'"

But this command relies on parent PID (.pid file) so if parent process was removed and child processes are still running it won't work.

So in my opinion accepted answer is the best. Or if you have installed use pkill

If using any monitoring service i.e monit like in the github link. Valid stop command would be:

gPPID=`ps -ef | /usr/bin/grep 'airflow scheduler' | /usr/bin/grep -v grep | awk '{print $2}' `; echo $gPPID; | xargs kill -15 && rm -f ${AIRFLOW_HOME}/airflow-scheduler.pid

score 0 · Answer 4 · edited Feb 19 '20 at 00:39

0

cd ~/airflow
cat airflow-scheduler.pid | xargs kill

edited Feb 19 '20 at 00:39

colidyre

4,170
12
37
53

answered Feb 19 '20 at 00:17

Will W

43
3

1

Code only answers are not as useful as answers that document the code or have an detailed explanation on why this code is the solution to the question. – RyanNerd Feb 19 '20 at 19:27

score 0 · Answer 5 · answered Apr 17 '20 at 05:53

0

To kill airflow webserver and scheduler, you can use below command

If you have supervisord configured for airflow, then stop it first

supervisorctl stop all

kill -9 `ps aux | grep airflow | awk '{print $2}'`

answered Apr 17 '20 at 05:53

Surya Venkatapathi

306
3
7

How to stop/kill airflow scheduler started in daemon mode

5 Answers5