5

I have a simple sidekiq job that I've set to run every 5 minutes using clockwork. What I'm worried about is that, if the job takes longer than 5 minutes, it will run again, which could cause some issues if it's running at the same time as the original job.

Is there a way to lock the clockwork so that it only runs if the previous running has completed?

Cisplatin
  • 2,860
  • 3
  • 36
  • 56
  • Sidekiq enterprise has unique job support if you don't want to purchase it, you can use a key in redis for the job and store it while job is running, and remove the key once job finishes. Only enqueue the job when key does not exist – Sam Mar 09 '17 at 21:10

4 Answers4

2

I've used the Sidekiq Unique Jobs gem to accomplish this. It has various strategies for preventing duplicate jobs.

Nick
  • 518
  • 1
  • 4
  • 16
1

Sidekiq Enterprise includes support for unique jobs. Unique jobs are defined with an option in the worker class:

# app/workers/my_worker.rb
class MyWorker
  include Sidekiq::Worker
  sidekiq_options unique_for: 10.minutes

  def perform(...)
  end
end

They must be enabled in the Sidekiq initializer as well:

# config/initializers/sidekiq.rb
Sidekiq::Enterprise.unique! unless Rails.env.test?

The time window only applies while the job is running, e.g., if it is unique for 10 minutes and the job completes in 6 minutes then Sidekiq will immediately allow another job to be enqueued; you don't have to wait 4 more minutes for the uniqueness lock to be dropped.

The uniqueness constraint is enforced across all Sidekiq Enterprise nodes.

You may need to adjust your retries options because a failed job will continue to hold the lock for as long as it is in a retry state. (even if it is not actively executing)

anothermh
  • 9,815
  • 3
  • 33
  • 52
0

Sidekiq allows you to query the queue through the API.

You could simply query the sidekiq queue when your task runs. Check to see if the queue and/or job is still there. If it is, do nothing. If it isn't, enqueue the job.

I don't know if your job sits in a separate queue, which would probably be best. That way you can just view the queue size and if it is 0 then start up another process.

Sidekiq API

Romuloux
  • 1,027
  • 1
  • 8
  • 24
-1

One option is to create a file with the process PID when the job starts, and delete it when the job ends, and abort the job at the beginning if the PID file already exists.

Although this particular part of my app isn't using sidekiq, the idea is the same. Here's how my app does it: https://github.com/WikiEducationFoundation/WikiEduDashboard/blob/master/lib/data_cycle/constant_update.rb

Sage Ross
  • 561
  • 5
  • 12
  • 4
    This does not guarantee the job is unique because Sidekiq can run concurrently across multiple nodes, and thus the job may or may not fire depending on which node ran the job. Since Sidekiq requires Redis, the better solution is to use Redis as the data store. An example would be in https://github.com/kenn/redis-mutex. – anothermh Mar 10 '17 at 21:35