Regularly purge stale Resque workers on Heroku?

Question

I've got Resque workers that typically shouldn't take longer than about 1-5 minutes to run, but frequently those workers will get "stuck" and go idle, clogging up workers and doing nothing.

So I'd like to regularly check for workers that have been running longer than X time and purge them. But I need to do this automatically, so I don't have to personally go in and manually clear them (Resque.workers.each {|w| w.unregister_worker}) every few hours.

This needs to work on Heroku.

Is the amount of time since a worker started a reliable indicator that it's stuck? I assume the 1-5 minutes is the duration of a job. But if jobs keep coming, a worker could run forever and not be "stuck." Yes? I ask because I'm having the same problem and I want a reliable way to clear the blockage. — Tim Scott, Aug 10 '13 at 22:22

score 6 · Accepted Answer · answered Jun 24 '13 at 11:26

6

Put this into a rake task:

allocated_time = 60 * 60 # 1 hour
Resque::WorkerRegistry.working.each do |worker|
  if (worker.started <=> Time.now - allocated_time) < 1
    worker.unregister
  end
end

Use heroku scheduler, you can set it to minimum of 10 minutes if that suites.

answered Jun 24 '13 at 11:26

Marc Greenstock

11,278
4
30
54

This appears to Resque 2 only, are people using this in production now? My impression from the README was that you shouldn't be using it yet. – opsb Aug 01 '13 at 08:40
It appears that worker.started will tell you when the worker was originally registered -- NOT how long it has been working on the current job. So this isn't the correct test to tell whether a worker is stuck or not. See [this answer](http://stackoverflow.com/a/21560463/629636) for a method of retrieving the start time of the current job. – Lachlan Cotter Mar 24 '14 at 02:12

score 1 · Answer 2 · answered Dec 13 '13 at 17:47

1

This worked for me to remove the specific workers running stale jobs. You could add it to a rake task.

Resque::Worker.working.each{|w| w.done_working }

answered Dec 13 '13 at 17:47

austen

3,314
3
25
26

score 0 · Answer 3 · answered Nov 14 '13 at 11:33

For Resque v1,

# lib/tasks/clear_stale_workers.rake
namespace :clear do
  desc 'Clearing stuck workers ...'
  task :stale_workers => :environment do
    Resque.workers.each do |w|
      w.unregister_worker unless w.started > 1.hour.ago
    end
  end
end

From the command line, rake clear:stale_workers

On Heroku, set the set the scheduler to run this Rake task.

Regularly purge stale Resque workers on Heroku?

3 Answers3