1

Is there a way to tell all workers to stop processing any new jobs for a particular queue (or all queues)?

We have an occasional issue where the systems that are needed become unavailable (power down, hardware failures, etc.), and it would be helpful to turn off the job processing in those queues until the systems come back online.

Instead of each job determining that all the subsystems are in a good/bad state, it would be helpful to have something that signals the queues to got dormant, and then bring them back alive when the subsystems are ready again.

A heavy handed way is to externally shut down all workers.

crispy
  • 11
  • 1

2 Answers2

1

The documentation mentions that you can use these signals to start/continue resque jobs.

Signals

Resque workers respond to a few different signals:

...

USR2 - Don't start to process any new jobs

CONT - Start to process new jobs again after a USR2

From: https://github.com/resque/resque/tree/cf0897709e857bd557dc7e407e282dc0f2b461df#signals

Community
  • 1
  • 1
Kashyap
  • 4,696
  • 24
  • 26
  • Thanks, I will have to consider this. Though it is awkward from a automation point of view. i.e. contacting each machine and issuing a kill signal for each process. – crispy Jun 16 '16 at 17:41
0

This gist, shows how resque can be monkey patched to stop picking up new jobs.

After pausing resque it might still be processing some jobs. Here is a rake task that waits 10 minutes for all jobs to finish, after 10 minutes if there are still jobs running they get killed:

  stop_at = Time.now

  until Resque.working.empty?
    Rails.logger.info "Waiting for #{Resque.info[:working].length} jobs to finish"

    sleep(30.seconds)

    if stop_at < 10.minutes.ago
      Rails.logger.info 'Killing all jobs that didnt finish in 10 minutes...'
      Resque.working.map(&:unregister_worker)
    end
  end
golfadas
  • 5,351
  • 3
  • 32
  • 39