1

I have an app that serves up many complex JSON pieces. Some of these could hit the database 1000 times. I pre-cache the results in our db such that when we change the underlying data we regenerate the JSON frag which then gets stored in the DB. I am using Sidekiq to manage this. One issue is that if I put this cache killing code into a Rails Model callback, I could potentially get multiple copies of the worker request at once. This is obviously not what I want. Is there a way in Sidekiq to say only add this to queue if it currently is not in the queue?

Here's an example of what I'm doing:

class ArcBackground
  include Sidekiq::Worker
  def perform(id)
    Menu.regenerateMenuJSON(id)
    Menu.regenerateMenuJSONFull(id)
  end
end
timpone
  • 19,235
  • 36
  • 121
  • 211
  • Add sidekiq config file with `:concurrency: 1` [link](https://github.com/mperham/sidekiq/wiki/Advanced-Options), or you are not talking about that? – zishe May 30 '14 at 02:10
  • thx - would that be concurrency for jobs? Would it be smart enough to know that those are the same jobs? sorry first time using Sidekiq. I just don't want a list of 10 identical items in the queue – timpone May 30 '14 at 02:11
  • It would be just one thread where all jobs performs, i'm not sure how to separate different jobs by different options. – zishe May 30 '14 at 02:13
  • 1
    ok thx - I think the UniqueJobs might do what I'm looking for: http://stackoverflow.com/questions/14713540/execute-only-one-of-many-duplicate-jobs-with-sidekiq – timpone May 30 '14 at 02:16
  • Sidekiq was designed to run jobs in concurrency which requires your jobs to be transactional and idempotent. So to answer your question, no it will not know that two jobs in a queue are identical. Here is the link to the best practices which explains what I mean: https://github.com/mperham/sidekiq/wiki/Best-Practices – Bill Watts Jun 20 '14 at 20:48
  • thx @BillWatts - I am not concerned about jobs being idempotent but rather have some long-running processes and the only one that matters is the last one such that if there's an intermediary one either in the queue or running, I'd like to be able to kill it. It sounds a bit more like an Actor pattern is what I'm looking for. thx – timpone Jun 21 '14 at 17:52
  • Only thing I can think of is isolate that job to a specific queue, check that queue to see if a job is already in process, if so kill it and add the new job to the queue. There are a couple of 3rd party gem that make it easy to query for a job status. https://github.com/mperham/sidekiq/wiki/Related-Projects – Bill Watts Jun 22 '14 at 01:31

0 Answers0