sidekiq on ubuntu failed to stop with upstart

Question

I installed an upstart service for sidekiq exactly as specified in it's wiki.

That said, I have an upstart script that creates a process with:

# skipped other insstructions
script
exec /bin/bash <<'EOT'
  source /home/me/.my_env
  cd /home/me
  exec bin/sidekiq -e production
EOT
end script

The service perfectly starts, but as one might see, sidekiq's PID is not equal to the PID that is known to upstart.

» initctl status sidekiq
# sidekiq start/running, process 16020
» ps axww|grep sidekiq
# 16181 ?        Sl     0:41 sidekiq 3.0.0 me[0 of 3 busy]

That configuration works for any other upstart script save for sidekiq:

» sudo initctl stop sidekiq
# sidekiq stop/waiting
» ps axww|grep sidekiq
# 16181 ?        Sl     0:45 sidekiq 3.0.0 me[0 of 3 busy]

It seems that initctl kills the bash process, that it treats as the job (16020), but underlying sidekiq continues to live:

» ps axww|grep  '16181\|16020'
# 16181 ?        Sl     0:45 sidekiq 3.0.0 me[0 of 3 busy]

What am I doing wrong?

score 1 · Answer 1 · answered Aug 26 '15 at 07:19

1

Sounds like sidekiq is deamonising or at least forking. So you need to follow http://upstart.ubuntu.com/cookbook/#expect, either find an option to sidekiq that runs in non-daemon mode, or add an appropriate expect statement.

answered Aug 26 '15 at 07:19

Douglas Leeder

2,745
18
15

Well, thanks for pointing me into right direction. The problem was deeper: I used complicated script inside `script` section, that created many processes during execution. Upstart seems to give up on that, so I finally added a `post-stop` section that gracefully shuts sidekiq down. – Aleksei Matiushkin Aug 26 '15 at 08:53

score 0 · Accepted Answer · edited Apr 13 '17 at 12:22

Credits to @douglas-leeder for pointing me into right direction.

sidekiq is neither demonising nor forking itself (unless -d option is explicitly specified.) But inside my exec script I called other processes, that upstart tried to count as my general process.

expect option does not help here, since actual sidekiq process was, say, 6th in the chain. Even worse: specifying expect fork leads to upstart hangup on stopping service, while expect daemon hungs it up on starting it. Refer to this answer on how to fight sidekiq hangup.

The solution I finally stuck to: one needs to define a post-stop section of upstart script, containing smth like:

post-stop script
   # ps-grep your PIDS
   # for sidekiq:
   SQ_PIDS=`ps -Ao pid,command | grep sidekiq | grep -v grep | awk '{ print $1 }' | sed 'N;s/\n/ /g'`
   sudo kill -USR1 $SQ_PIDS
   sleep 3
   sudo kill -9 $SQ_PIDS
   sudo rm sidekiq.pid 2>&1 >/dev/null
end script

The above will try to gracefully shut down sidekiq and kill it after all. Not the most elegant solution, but it works.

sidekiq on ubuntu failed to stop with upstart

2 Answers2