1

Let's say we have 100 objects to send in Sidekiq to process some data. A rake is written which runs every 24 hours and picks up all 100 object where the object has column sidekiq_processed as false and send it into Sidekiq queue for processing. After Sidekiq has processed data sidekiq_processed column of the object is updated as true so that next time when the rake is scheduled that object is not picked up.

The problem I'm facing is some objects after say 20 days still have sidekiq_processed as false. I have also implemented logging inside perform function to note each event/state in which my objects are.

Let's say there were 20 objects left which didn't get processed. Their logs were checked and no logs of these objects were found. After triggering new.perform on the 21st day all objects were finally processed.

I don't want to trigger them manually is there a way we could check why Sidekiq didn't consume my queue objects.

task :process_some_objects => :environment do
    Object.where(sidekiq_processed: false).each do |object|
        Worker.perform_async(object.id)
    end
end
Harsimar Sandhu
  • 107
  • 6
  • Make sure the sidekiq is up, check sidekiq log, use sidekiq web ui and find for enqueued but not executed jobs, check queues - there are number of possible explanations to this. – Marek Lipka Oct 10 '18 at 07:05
  • Sidekiq is up 24 * 7 no downtime. The job ran minimum 20 times as rake was scheduled every 24 hours. Enqueuing might not be a problem here as Sidekiq also retries to finish the failed job plus job was executed every day for 20 days. – Harsimar Sandhu Oct 10 '18 at 07:16
  • Well then maybe your worker isn't written correctly? Does it work if you run `Worker.new.perform(object.id)` from console? – Marek Lipka Oct 10 '18 at 07:30
  • yes as I said in the post some objects get processed with perform_async for some I have to do new.perform. assume all objects are identical and pass all conditions in perform. – Harsimar Sandhu Oct 10 '18 at 07:32
  • 1
    Instead of a boolean sidekiq_processed, use a nullable sidekiq_jid column which stores the JID associated with the element. Set it to "" when done. Null == not processed. JID == in progress, "" == done. – Mike Perham Oct 10 '18 at 20:23
  • Then you can grep the Sidekiq logs for the JID to see associated logging. The JID is returned by perform_async. – Mike Perham Oct 10 '18 at 20:24

0 Answers0