1

I've got a django app running on webfaction, and a couple of days ago I upgraded a package that also upgraded django to version 2.1 (from version 2.0). Shortly thereafter, celery stopped working, and when I tried to restart it in supervisor I got the following error:

http://127.0.0.1:12637 refused connection

When I checked the supervisor logs, I saw the following: 

2018-10-06 21:56:58,356 WARN received SIGHUP indicating restart request
2018-10-06 21:56:58,417 INFO waiting for accconsole_celery, catdemo, accconsole_celerybeat to die
2018-10-06 21:56:58,418 INFO exited: catdemo (terminated by SIGTERM; not expected)
2018-10-06 21:56:58,432 WARN received SIGTERM indicating exit request
2018-10-06 21:56:58,590 INFO stopped: accconsole_celerybeat (terminated by SIGHUP)
2018-10-06 21:57:01,604 INFO waiting for accconsole_celery to die
2018-10-06 21:57:04,721 INFO waiting for accconsole_celery to die
2018-10-06 21:57:07,729 INFO waiting for accconsole_celery to die
2018-10-06 21:57:10,736 INFO waiting for accconsole_celery to die
2018-10-06 21:57:13,740 INFO waiting for accconsole_celery to die
2018-10-06 21:57:16,747 INFO waiting for accconsole_celery to die
2018-10-06 21:57:19,752 INFO waiting for accconsole_celery to die
2018-10-06 21:57:22,758 INFO waiting for accconsole_celery to die
2018-10-06 21:57:25,762 INFO waiting for accconsole_celery to die
2018-10-06 21:57:28,766 INFO waiting for accconsole_celery to die
2018-10-06 21:57:31,769 INFO waiting for accconsole_celery to die
2018-10-06 21:57:34,774 INFO waiting for accconsole_celery to die
2018-10-06 21:57:37,778 INFO waiting for accconsole_celery to die
2018-10-06 21:57:40,781 INFO waiting for accconsole_celery to die
2018-10-06 21:57:43,784 INFO waiting for accconsole_celery to die

So I imagine some process isn't properly shutting off, but I can't seem to force it to quit. Every time I try: ps aux|grep 'celery worker' I get a worker with a different process number, as so:

acounsel 11386  0.0  0.0 112704   980 pts/1    S+   20:48   0:00 grep --color=auto celery worker
acounsel 11446  0.0  0.0 112704   980 pts/1    S+   20:49   0:00 grep --color=auto celery worker

So when I try to kill them, I get responses like this: -bash: kill: (11446) - No such process

pkill -9 -f 'celery worker' and the longer ps auxww | grep 'celery worker' | awk '{print $2}' | xargs kill -9 similarly don't seem to do anything. I also can't run celery commands like celery -A django_aconsole status or celery -A django_aconsole purge, it just hangs and doesn't seem to do anything.

Any ideas how I can shut down these processes so I can restart celery?

Samer
  • 156
  • 7
  • Supervisor will restart celery for you every time you shut it down. If you get a `refused connection` from supervisor, you are doing something wrong. Is it the correct port? Did you try running `supervisorctl` as root? – Håken Lid Oct 08 '18 at 21:55
  • The port is definitely right, but I don't know how to run supervisorctl as root on webfaction. Thanks for your help! – Samer Oct 09 '18 at 04:16

1 Answers1

1

It was a simple problem; supervisord had stopped running, and so supervisorctl couldn't run without it. Just needed to run supervisord first and everything worked right as rain.

Samer
  • 156
  • 7