Sidekiq job in initializer is fired twice

Question

I have a rails web app and I´m integration a backgroud process with sidekiq. Basic integration is working fine.

The point is that I need to run the background process continuously in a loop. According to another post I have created a loop that fired the worker in an initializer. The code:

sidekiq.rb

Thread.new do
  loop do    
    puts "Sidekiq.Initializer-->before perform worker"
    MyWorker.perform_async()    
    puts "Sidekiq.Initializer-->after perform worker and before sleep"
    sleep(1.minutes)
    puts "Sidekiq.Initializer-->after sleep"
  end
end

However, this initializer is running twice. First when the rails server is started up and second when the sidekiq server is started up. How can I do to have this functionality running just once.

Remember that Sidekiq will try to execute your job at least once and you should write your logic to account for this. This is mentioned in best practices as well: https://github.com/mperham/sidekiq/wiki/Best-Practices — Josh Brody, Apr 23 '20 at 16:51
Thanks for the reminder. This is actually my question. How can I control it in case I need to run the worker in a loop, without duplicating the thread and the jobs in the initializer? Is there a better sidekiq way to accomplish this? — Rober, Apr 23 '20 at 16:57
It really depends on what your worker is doing. If you could post the contents I could give you a better idea as to an approach. — Josh Brody, Apr 23 '20 at 17:41
Well, I don´t think to post the whole code is feasible. Actually, this worker run a method that makes a complex synchronization with a third party system. This operation can take more than an hour. — Rober, Apr 23 '20 at 18:04

score 0 · Answer 1 · answered Apr 23 '20 at 16:04

0

Sounds like you have some strangeness with load ordering, but a general pattern for this would be to wrap it in a method, then call that method from an initialiser if you want it to be called on Rails start. I wouldn't think that the Sidekiq server would be able to run it then.

answered Apr 23 '20 at 16:04

CJW

332
1
14

Sorry don´t get your point. The code that I show above is in an initializer. The point is that this initializer is been called twice, so when rails starts and the sidekiq server starts. What is the difference between wrap the code in a method, if the method will be called in the initializer, and the initializer is called twice? – Rober Apr 23 '20 at 16:47
Yep, sorry I guess the above doesn't really make sense based on your question, I was trying to hint at the fact that if you trigger the method after all the loading is done you won't have this problem. This person had a similar problem. https://stackoverflow.com/questions/17812784/how-can-i-run-a-continuous-background-job-with-sidekiq/17980713 A quick note about tools that may be useful for you: https://github.com/moove-it/sidekiq-scheduler – CJW Apr 24 '20 at 11:17

score 0 · Answer 2 · answered Apr 23 '20 at 18:56

The more obvious and ideal way would be to write a value to the database and check if it exists in your worker before you execute it.

These probably won't scale like you need them to, but because I don't know about your architecture I'll give you some leads so you can explore your own path:

Inside config/application.rb:

def ran_special_job?
  @ran_special_job < 5.minutes.ago
end

def ran_special_job_at=(val)
  @ran_special_job = val
end

In your job itself, short-circuit it:

def perform
  return if Rails.application.ran_special_job?
  Rails.application.ran_special_job_at = Time.now
  puts "I am running"
end

Another way, your worker:

def perform
  return if Rails.application.ran_special_job?
  Rails.application.ran_special_job!
  puts "I am running"
end

and in your config/application.rb

def ran_special_job?
  return false unless File.exists?("special_job.txt")
  DateTime.parse(File.read("special_job.txt")) < 5.minutes.ago
end

def ran_special_job!
  File.open("special_job.txt", "w") { |file| file.write Time.now }
end

This method isn't ideal if you're on Heroku or some other service that doesn't provide a persistent filesystem, but you get the idea.

Sidekiq job in initializer is fired twice

2 Answers2