2

I'm running into an extremely strange bug with Resque.

  • I got 5 workers running, with one queue, waiting for jobs.
  • I got 1 type of Resque worker : MediaAnalyzer

And here is the strangest thing in the world : sometimes (yeah, SOMETIMES), resque can't find the class MediaAnalyzer.

Why sometimes ? Because sometimes job is processed without any problem at all. Yet some other times I got uninitialized constant MediaAnalyzer. Even stranger, if I retry the job via resque-web, after a couple of times, the worker is finally found and performed...

All workers are started in the same way with the same command (except for PID and log paths) :

su -c "cd /myapp/current; bundle exec rake environment resque:work RAILS_ENV=production QUEUES=* BACKGROUND=yes PIDFILE=/myapp/current/tmp/pids/resque_worker.1.pid 2>&1 >> /myapp/shared/log/resque_worker.1.log" - rails

Note : even when started manually on the console in foreground mode I keep getting the same bug.

I even tried to manually load the class via the resque:setup task :

task "resque:setup" => :environment do
  require Rails.root.join("app/workers/media_analyzer").to_s
end

This cause no error, I can after this require use everything in the class, I can output stuff so this rake task is actually called... but resque keeps failing at loading this class later.

The biggest difficulty is that this bug is really genuinely random !

Here is a pastebin of the exception backtrace : http://pastebin.com/jy5UakB8

Tried with resque 2.0, same issue.

If you know what is happening you're a genius !

PofMagicfingers
  • 215
  • 2
  • 14

3 Answers3

1

The issue you experience to be random can easily be caused by the Rails/Ruby autoloading. The order files get loaded are unguessable sometimes. Combined with using the wrong file name for your class or putting it into the wrong directory may lead to this issue.

You don't get the exception if you explicitly require the file your class is defined in before referencing the class itself. If you reference first Rails try to find the according file in the load path. If it cannot find it, it raises.

Please double check your file names and have a look at the $LOAD_PATH variable. app/workers should be included.

An easy test to start with is to run rails console and simply access MediaAnalyzer.

iltempo
  • 15,718
  • 8
  • 61
  • 72
  • I got no problems accessing ```MediaAnalyzer``` via the console :/ ```app/workers``` is in the load path... – PofMagicfingers Apr 18 '14 at 10:15
  • But even when requiring (successfully) the file via ```resque:setup```, resque keeps "randomly" failing at loading the class. Even if I can access the class in resque:setup without any problem. – PofMagicfingers Apr 18 '14 at 10:19
1

Okay guys, I fixed the issue. And it was a really stupid issue.

I got 2 staging for this application, beta and production. The actual issue was that the MediaAnalyzer is not anymore present in the beta application... And my workers from beta were performing the jobs instead of the workers from the production application.

I didn't pay enough attention to the exception log, it was saying that myapp_beta/ in the backtrace.

The issue is simple to resolve, I just had to namespace the redis/resque with :

 Resque.redis.namespace = "resque:myapp_#{Rails.env}"

And add this environment variable to the worker command :

RESQUE_NAMESPACE=resque:myapp_production

(or RESQUE_NAMESPACE=resque:myapp_beta for beta obviously)

That's why it was random, it was depending on which worker was catching the job, one from the beta or one from the production.

PofMagicfingers
  • 215
  • 2
  • 14
1

If you're using ActiveJob and noticed this, what I found to fix the issue is just to restart your Redis server. I'm not sure why this works quite yet but everything started working right after I did that and killed off all the workers.

baweaver
  • 304
  • 1
  • 12