Is there way to run code before Sidekiq is restarted in the middle of a job?

Question

I have a Sidekiq job that runs every 4 minutes.

This job checks if the current code block is being executed before executing the code again

process = ProcessTime.where("name = 'ad_queue_process'").first

# Return if job is running
return if process.is_running == true

If Sidekiq restarts midway through the code block, code that updates the status of the job never runs

# Done running, update the process times and allow it to be ran again
process.update_attributes(is_running: false, last_execution_time: Time.now)

Which leads the the Job never running unless i run an update statement to set is_running = false

Is there any way to execute code before Sidekiq is restarted?

Answer 1

Update:

• Thanks to @Aaron, and following our discussion (comments below), the ensure block (which is executed by the forked worker-threads) can only be ran for a few unguaranteed milliseconds before the main-thread forcefully terminates these worker-threads, in order for the main-thread to do some "cleanup" up the exception stack, in order to avoid getting SIGKILL-ed by Heroku. Therefore, make sure that your ensure code should be really fast!

TL;DR:

def perform(*args)
  # your code here
ensure
  process.update_attributes(is_running: false, last_execution_time: Time.now)
end

• The ensure above is always called regardless if the method "succeeded" or an Exception is raised. I tested this: see this repl code, and click "Run"

• In other words, this is always called even on a SignalException, even if the signal is SIGTERM (gracefully shutdown signal), but ONLY EXCEPT on SIGKILL (force unrescueable shutdown). You can verify this behaviour by checking my repl code, and then change Process.kill('TERM', Process.pid) to Process.kill('KILL', Process.pid), and then click "run" again (you'll notice that the puts won't be called)

• Looking at Heroku docs, I quote:

  When Heroku is going to shut down a dyno (for a restart or a new deploy, etc.), it first sends a SIGTERM signal to the processes in the dyno.

  After Heroku sends SIGTERM to your application, it will wait a few seconds and then send SIGKILL to force it to shut down, even if it has not finished cleaning up. In this example, the ensure block does not get called at all, the program simply exits

  ... which means that the ensure block will be called because it's a SIGTERM and not a SIGKILL, only except if the shutting down takes a looong time, which may due to (some reasons I could think of ATM):

  • Something inside your perform code (or any ruby code in the stack; even gems) that also rescued the SignalException, or even rescued the root Exception class because SignalException is a subclass of Exception) but takes a long time cleaning up (i.e. cleaning up connections to DB or something, or I/O stuff that hangs your application)

  • Or, your own ensure block above takes a looong time. I.E when doing the process.update_attributes(...), for some reason the DB temporary hangs / network delay or timeout, then that update might not succeed at all! and will ran out of time, of which from my quote above, after a few seconds after the SIGTERM, the application will be forced to be stopped by Heroku sending a SIGKILL.

... which all means that my solution is still not fully reliable, but should work under normal situations

Answer 2

Handle sidekiq shutdown exception

class SomeWorker
  include Sidekiq::Worker

  sidekiq_options queue: :default

  def perform(params)
    ...

  rescue Sidekiq::Shutdown
    SomeWorker.perform_async(params)
  end
end