Resque: don't enqueue redundant jobs?

Question

I have a set of Resque queues, arranged by priority. For all of the jobs on the lowest-priority queue, they have no arguments, and a later run will supersede any previous runs -- e.g. they are syncing data with an external service. So it makes no sense to create a backlog with multiple copies of these jobs -- running it three times in a row adds no value.

But during busy periods, we do get a backlog (on this queue only) since the workers are busy with higher-priority jobs.

Can I prevent this backlog by stopping redundant jobs from being enqueued in the first place?

To prevent the backlog, first somehow you need to track that the same job has already enqueued. Because resque & redis by default doesn't provide such functionality of uniqueness. — Ketan Doshi, Sep 11 '17 at 08:02

score 0 · Accepted Answer · answered Oct 13 '17 at 13:35

Yes, there are Gems that are available to do this. (AFAICT, they work by hashing the job class with its list of arguments and maintaining a set of "keys already enqueued" separately in Redis.)

See https://github.com/resque/resque-loner and https://github.com/neighborland/resque_solo.

Resque: don't enqueue redundant jobs?

1 Answers1